Jul 29 14:34:57.834527 ip-10-0-136-196 systemd[1]: Starting Kubernetes Kubelet...
Jul 29 14:34:58.239001 ip-10-0-136-196 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 14:34:58.239001 ip-10-0-136-196 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 29 14:34:58.239001 ip-10-0-136-196 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 14:34:58.239001 ip-10-0-136-196 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 29 14:34:58.239001 ip-10-0-136-196 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 14:34:58.239727 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.239061    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 29 14:34:58.244858 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244836    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 14:34:58.244858 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244855    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 14:34:58.244858 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244859    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 14:34:58.244858 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244862    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 14:34:58.244858 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244865    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 14:34:58.245055 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244868    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 14:34:58.245055 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244871    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 14:34:58.245055 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244874    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 14:34:58.245055 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244877    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 14:34:58.245055 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244879    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 14:34:58.245055 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244882    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 14:34:58.245055 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244885    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 14:34:58.245055 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244887    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 14:34:58.245055 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244890    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 14:34:58.245055 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244892    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 14:34:58.245055 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244896    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 14:34:58.245055 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244899    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 14:34:58.245055 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244901    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 14:34:58.245055 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244904    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 14:34:58.245055 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244906    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 14:34:58.245055 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244909    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 14:34:58.245055 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244912    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 14:34:58.245055 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244915    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 14:34:58.245055 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244918    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 14:34:58.245055 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244921    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 14:34:58.245558 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244923    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 14:34:58.245558 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244926    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 14:34:58.245558 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244929    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 14:34:58.245558 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244932    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 14:34:58.245558 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244935    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 14:34:58.245558 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244937    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 14:34:58.245558 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244941    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 14:34:58.245558 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244943    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 14:34:58.245558 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244946    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 14:34:58.245558 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244949    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 14:34:58.245558 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244952    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 14:34:58.245558 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244954    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 14:34:58.245558 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244957    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 14:34:58.245558 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244961    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 14:34:58.245558 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244964    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 14:34:58.245558 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244968    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 14:34:58.245558 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244972    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 14:34:58.245558 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244976    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 14:34:58.245558 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244978    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 14:34:58.245558 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244981    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 14:34:58.246039 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244985    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 14:34:58.246039 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244988    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 14:34:58.246039 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244990    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 14:34:58.246039 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244993    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 14:34:58.246039 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244996    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 14:34:58.246039 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.244998    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 14:34:58.246039 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245001    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 14:34:58.246039 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245003    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 14:34:58.246039 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245006    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 14:34:58.246039 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245008    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 14:34:58.246039 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245012    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 14:34:58.246039 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245014    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 14:34:58.246039 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245017    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 14:34:58.246039 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245020    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 14:34:58.246039 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245022    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 14:34:58.246039 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245025    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 14:34:58.246039 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245028    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 14:34:58.246039 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245030    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 14:34:58.246039 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245033    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 14:34:58.246039 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245035    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 14:34:58.246537 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245038    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 14:34:58.246537 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245040    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 14:34:58.246537 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245044    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 14:34:58.246537 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245048    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 14:34:58.246537 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245051    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 14:34:58.246537 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245054    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 14:34:58.246537 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245057    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 14:34:58.246537 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245060    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 14:34:58.246537 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245064    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 14:34:58.246537 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245069    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 14:34:58.246537 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245072    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 14:34:58.246537 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245075    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 14:34:58.246537 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245078    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 14:34:58.246537 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245080    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 14:34:58.246537 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245083    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 14:34:58.246537 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245085    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 14:34:58.246537 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245088    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 14:34:58.246537 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245090    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 14:34:58.246981 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245093    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 14:34:58.246981 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245096    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 14:34:58.246981 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245099    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 14:34:58.246981 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245101    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 14:34:58.246981 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245104    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 14:34:58.246981 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245106    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 14:34:58.246981 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245109    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 14:34:58.246981 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245111    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 14:34:58.246981 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245114    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 14:34:58.246981 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245117    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 14:34:58.246981 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245119    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 14:34:58.246981 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245122    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 14:34:58.246981 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245124    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 14:34:58.246981 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245127    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 14:34:58.246981 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245144    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 14:34:58.246981 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245149    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 14:34:58.246981 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245861    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 14:34:58.246981 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245868    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 14:34:58.246981 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245871    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 14:34:58.246981 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245874    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 14:34:58.247488 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245877    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 14:34:58.247488 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245880    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 14:34:58.247488 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245882    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 14:34:58.247488 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245885    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 14:34:58.247488 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245888    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 14:34:58.247488 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245890    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 14:34:58.247488 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245893    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 14:34:58.247488 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245895    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 14:34:58.247488 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245898    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 14:34:58.247488 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245901    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 14:34:58.247488 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245904    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 14:34:58.247488 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245906    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 14:34:58.247488 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245909    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 14:34:58.247488 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245912    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 14:34:58.247488 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245915    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 14:34:58.247488 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245918    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 14:34:58.247488 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245921    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 14:34:58.247488 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245924    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 14:34:58.247488 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245926    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 14:34:58.247488 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245929    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 14:34:58.247990 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245931    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 14:34:58.247990 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245934    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 14:34:58.247990 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245936    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 14:34:58.247990 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245939    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 14:34:58.247990 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245941    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 14:34:58.247990 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245944    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 14:34:58.247990 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245946    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 14:34:58.247990 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245949    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 14:34:58.247990 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245951    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 14:34:58.247990 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245954    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 14:34:58.247990 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245957    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 14:34:58.247990 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245960    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 14:34:58.247990 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245962    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 14:34:58.247990 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245965    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 14:34:58.247990 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245968    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 14:34:58.247990 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245971    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 14:34:58.247990 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245973    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 14:34:58.247990 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245976    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 14:34:58.247990 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245978    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 14:34:58.247990 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245981    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 14:34:58.248498 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245985    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 14:34:58.248498 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245989    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 14:34:58.248498 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245992    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 14:34:58.248498 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245996    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 14:34:58.248498 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.245999    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 14:34:58.248498 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246001    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 14:34:58.248498 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246005    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 14:34:58.248498 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246007    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 14:34:58.248498 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246010    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 14:34:58.248498 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246013    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 14:34:58.248498 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246016    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 14:34:58.248498 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246019    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 14:34:58.248498 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246021    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 14:34:58.248498 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246024    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 14:34:58.248498 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246026    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 14:34:58.248498 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246029    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 14:34:58.248498 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246032    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 14:34:58.248498 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246034    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 14:34:58.248498 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246037    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 14:34:58.248977 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246039    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 14:34:58.248977 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246042    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 14:34:58.248977 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246045    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 14:34:58.248977 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246048    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 14:34:58.248977 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246052    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 14:34:58.248977 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246055    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 14:34:58.248977 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246057    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 14:34:58.248977 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246060    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 14:34:58.248977 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246063    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 14:34:58.248977 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246066    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 14:34:58.248977 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246068    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 14:34:58.248977 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246071    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 14:34:58.248977 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246073    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 14:34:58.248977 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246076    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 14:34:58.248977 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246078    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 14:34:58.248977 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246083    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 14:34:58.248977 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246085    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 14:34:58.248977 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246088    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 14:34:58.248977 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246090    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 14:34:58.248977 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246093    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 14:34:58.249506 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246096    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 14:34:58.249506 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246099    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 14:34:58.249506 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246101    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 14:34:58.249506 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246104    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 14:34:58.249506 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246106    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 14:34:58.249506 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246109    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 14:34:58.249506 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246111    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 14:34:58.249506 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246114    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 14:34:58.249506 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246117    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 14:34:58.249506 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246121    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 14:34:58.249506 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246124    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 14:34:58.249506 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246127    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 14:34:58.249506 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246146    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 14:34:58.249506 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246149    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 14:34:58.249506 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246152    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 14:34:58.249506 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246156    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 14:34:58.249506 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246549    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 14:34:58.249506 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246553    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 14:34:58.249506 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246557    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 14:34:58.249963 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246559    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 14:34:58.249963 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246562    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 14:34:58.249963 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246565    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 14:34:58.249963 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246568    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 14:34:58.249963 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246571    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 14:34:58.249963 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246573    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 14:34:58.249963 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246576    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 14:34:58.249963 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246578    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 14:34:58.249963 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246582    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 14:34:58.249963 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246585    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 14:34:58.249963 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246588    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 14:34:58.249963 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246591    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 14:34:58.249963 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246593    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 14:34:58.249963 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246596    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 14:34:58.249963 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246599    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 14:34:58.249963 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246602    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 14:34:58.249963 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246605    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 14:34:58.249963 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246607    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 14:34:58.249963 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246610    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 14:34:58.249963 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246613    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 14:34:58.250451 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246615    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 14:34:58.250451 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246618    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 14:34:58.250451 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246620    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 14:34:58.250451 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246623    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 14:34:58.250451 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246625    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 14:34:58.250451 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246628    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 14:34:58.250451 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246630    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 14:34:58.250451 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246633    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 14:34:58.250451 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246635    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 14:34:58.250451 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246638    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 14:34:58.250451 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246641    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 14:34:58.250451 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246644    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 14:34:58.250451 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246646    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 14:34:58.250451 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246648    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 14:34:58.250451 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246652    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 14:34:58.250451 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246654    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 14:34:58.250451 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246657    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 14:34:58.250451 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246659    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 14:34:58.250451 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246662    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 14:34:58.250451 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246664    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 14:34:58.250963 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246667    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 14:34:58.250963 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246669    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 14:34:58.250963 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246673    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 14:34:58.250963 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246676    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 14:34:58.250963 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246679    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 14:34:58.250963 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246682    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 14:34:58.250963 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246686    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 14:34:58.250963 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246689    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 14:34:58.250963 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246691    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 14:34:58.250963 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246694    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 14:34:58.250963 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246697    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 14:34:58.250963 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246700    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 14:34:58.250963 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246702    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 14:34:58.250963 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246705    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 14:34:58.250963 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246707    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 14:34:58.250963 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246710    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 14:34:58.250963 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246712    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 14:34:58.250963 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246715    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 14:34:58.250963 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246717    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 14:34:58.251433 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246720    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 14:34:58.251433 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246722    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 14:34:58.251433 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246725    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 14:34:58.251433 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246728    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 14:34:58.251433 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246730    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 14:34:58.251433 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246733    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 14:34:58.251433 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246735    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 14:34:58.251433 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246737    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 14:34:58.251433 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246740    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 14:34:58.251433 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246743    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 14:34:58.251433 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246745    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 14:34:58.251433 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246747    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 14:34:58.251433 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246750    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 14:34:58.251433 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246752    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 14:34:58.251433 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246755    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 14:34:58.251433 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246757    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 14:34:58.251433 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246760    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 14:34:58.251433 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246763    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 14:34:58.251433 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246765    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 14:34:58.251433 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246768    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 14:34:58.251976 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246772    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 14:34:58.251976 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246774    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 14:34:58.251976 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246777    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 14:34:58.251976 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246779    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 14:34:58.251976 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246781    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 14:34:58.251976 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246785    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 14:34:58.251976 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246788    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 14:34:58.251976 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246791    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 14:34:58.251976 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246793    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 14:34:58.251976 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246795    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 14:34:58.251976 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246798    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 14:34:58.251976 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246800    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 14:34:58.251976 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246803    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 14:34:58.251976 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246805    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 14:34:58.251976 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246808    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 14:34:58.251976 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246812    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 14:34:58.251976 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.246815    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 14:34:58.251976 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.246886    2563 flags.go:64] FLAG: --address="0.0.0.0"
Jul 29 14:34:58.251976 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.246893    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 29 14:34:58.251976 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.246909    2563 flags.go:64] FLAG: --anonymous-auth="true"
Jul 29 14:34:58.252472 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.246914    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 29 14:34:58.252472 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.246918    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 29 14:34:58.252472 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.246921    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 29 14:34:58.252472 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.246925    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 29 14:34:58.252472 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.246930    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 29 14:34:58.252472 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.246933    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 29 14:34:58.252472 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.246936    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 29 14:34:58.252472 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.246940    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 29 14:34:58.252472 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.246943    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 29 14:34:58.252472 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.246946    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 29 14:34:58.252472 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.246949    2563 flags.go:64] FLAG: --cgroup-root=""
Jul 29 14:34:58.252472 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.246954    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 29 14:34:58.252472 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.246957    2563 flags.go:64] FLAG: --client-ca-file=""
Jul 29 14:34:58.252472 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.246961    2563 flags.go:64] FLAG: --cloud-provider="external"
Jul 29 14:34:58.252472 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.246964    2563 flags.go:64] FLAG: --cluster-dns="[]"
Jul 29 14:34:58.252472 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.246971    2563 flags.go:64] FLAG: --cluster-domain=""
Jul 29 14:34:58.252472 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.246974    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 29 14:34:58.252472 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.246977    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 29 14:34:58.252472 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.246982    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 29 14:34:58.252472 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.246986    2563 flags.go:64] FLAG: --container-log-max-files="5"
Jul 29 14:34:58.252472 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.246990    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 29 14:34:58.252472 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.246993    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 29 14:34:58.252472 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.246996    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 29 14:34:58.253039 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.246999    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 29 14:34:58.253039 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247002    2563 flags.go:64] FLAG: --contention-profiling="false"
Jul 29 14:34:58.253039 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247005    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 29 14:34:58.253039 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247008    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 29 14:34:58.253039 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247011    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 29 14:34:58.253039 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247015    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 29 14:34:58.253039 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247019    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 29 14:34:58.253039 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247022    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 29 14:34:58.253039 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247024    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 29 14:34:58.253039 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247027    2563 flags.go:64] FLAG: --enable-load-reader="false"
Jul 29 14:34:58.253039 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247030    2563 flags.go:64] FLAG: --enable-server="true"
Jul 29 14:34:58.253039 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247033    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 29 14:34:58.253039 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247040    2563 flags.go:64] FLAG: --event-burst="100"
Jul 29 14:34:58.253039 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247043    2563 flags.go:64] FLAG: --event-qps="50"
Jul 29 14:34:58.253039 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247046    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 29 14:34:58.253039 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247049    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 29 14:34:58.253039 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247052    2563 flags.go:64] FLAG: --eviction-hard=""
Jul 29 14:34:58.253039 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247056    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 29 14:34:58.253039 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247059    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 29 14:34:58.253039 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247062    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 29 14:34:58.253039 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247066    2563 flags.go:64] FLAG: --eviction-soft=""
Jul 29 14:34:58.253039 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247069    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 29 14:34:58.253039 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247071    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 29 14:34:58.253039 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247075    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 29 14:34:58.253039 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247078    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 29 14:34:58.253663 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247081    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 29 14:34:58.253663 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247084    2563 flags.go:64] FLAG: --fail-swap-on="true"
Jul 29 14:34:58.253663 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247087    2563 flags.go:64] FLAG: --feature-gates=""
Jul 29 14:34:58.253663 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247091    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 29 14:34:58.253663 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247094    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 29 14:34:58.253663 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247097    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 29 14:34:58.253663 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247100    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 29 14:34:58.253663 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247103    2563 flags.go:64] FLAG: --healthz-port="10248"
Jul 29 14:34:58.253663 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247106    2563 flags.go:64] FLAG: --help="false"
Jul 29 14:34:58.253663 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247108    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-136-196.ec2.internal"
Jul 29 14:34:58.253663 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247111    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 29 14:34:58.253663 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247114    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 29 14:34:58.253663 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247117    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 29 14:34:58.253663 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247120    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 29 14:34:58.253663 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247124    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 29 14:34:58.253663 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247126    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 29 14:34:58.253663 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247145    2563 flags.go:64] FLAG: --image-service-endpoint=""
Jul 29 14:34:58.253663 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247149    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 29 14:34:58.253663 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247154    2563 flags.go:64] FLAG: --kube-api-burst="100"
Jul 29 14:34:58.253663 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247157    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 29 14:34:58.253663 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247160    2563 flags.go:64] FLAG: --kube-api-qps="50"
Jul 29 14:34:58.253663 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247163    2563 flags.go:64] FLAG: --kube-reserved=""
Jul 29 14:34:58.253663 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247166    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 29 14:34:58.253663 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247169    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 29 14:34:58.254308 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247172    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 29 14:34:58.254308 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247175    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 29 14:34:58.254308 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247178    2563 flags.go:64] FLAG: --lock-file=""
Jul 29 14:34:58.254308 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247181    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 29 14:34:58.254308 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247184    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 29 14:34:58.254308 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247187    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 29 14:34:58.254308 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247192    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 29 14:34:58.254308 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247196    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 29 14:34:58.254308 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247199    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 29 14:34:58.254308 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247202    2563 flags.go:64] FLAG: --logging-format="text"
Jul 29 14:34:58.254308 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247206    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 29 14:34:58.254308 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247209    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 29 14:34:58.254308 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247212    2563 flags.go:64] FLAG: --manifest-url=""
Jul 29 14:34:58.254308 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247215    2563 flags.go:64] FLAG: --manifest-url-header=""
Jul 29 14:34:58.254308 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247219    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 29 14:34:58.254308 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247223    2563 flags.go:64] FLAG: --max-open-files="1000000"
Jul 29 14:34:58.254308 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247227    2563 flags.go:64] FLAG: --max-pods="110"
Jul 29 14:34:58.254308 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247230    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 29 14:34:58.254308 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247233    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 29 14:34:58.254308 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247236    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 29 14:34:58.254308 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247238    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 29 14:34:58.254308 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247241    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 29 14:34:58.254308 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247244    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 29 14:34:58.254308 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247247    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 29 14:34:58.254308 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247255    2563 flags.go:64] FLAG: --node-status-max-images="50"
Jul 29 14:34:58.254943 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247258    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 29 14:34:58.254943 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247261    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 29 14:34:58.254943 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247264    2563 flags.go:64] FLAG: --pod-cidr=""
Jul 29 14:34:58.254943 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247267    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 29 14:34:58.254943 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247272    2563 flags.go:64] FLAG: --pod-manifest-path=""
Jul 29 14:34:58.254943 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247274    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 29 14:34:58.254943 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247278    2563 flags.go:64] FLAG: --pods-per-core="0"
Jul 29 14:34:58.254943 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247280    2563 flags.go:64] FLAG: --port="10250"
Jul 29 14:34:58.254943 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247284    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 29 14:34:58.254943 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247286    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ab38b9689f8f1ff4"
Jul 29 14:34:58.254943 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247290    2563 flags.go:64] FLAG: --qos-reserved=""
Jul 29 14:34:58.254943 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247293    2563 flags.go:64] FLAG: --read-only-port="10255"
Jul 29 14:34:58.254943 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247295    2563 flags.go:64] FLAG: --register-node="true"
Jul 29 14:34:58.254943 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247298    2563 flags.go:64] FLAG: --register-with-taints=""
Jul 29 14:34:58.254943 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247302    2563 flags.go:64] FLAG: --registry-burst="10"
Jul 29 14:34:58.254943 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247305    2563 flags.go:64] FLAG: --registry-qps="5"
Jul 29 14:34:58.254943 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247308    2563 flags.go:64] FLAG: --reserved-cpus=""
Jul 29 14:34:58.254943 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247312    2563 flags.go:64] FLAG: --reserved-memory=""
Jul 29 14:34:58.254943 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247315    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 29 14:34:58.254943 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247318    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 29 14:34:58.254943 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247321    2563 flags.go:64] FLAG: --rotate-certificates="false"
Jul 29 14:34:58.254943 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247324    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 29 14:34:58.254943 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247327    2563 flags.go:64] FLAG: --runonce="false"
Jul 29 14:34:58.254943 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247330    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 29 14:34:58.254943 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247333    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 29 14:34:58.255607 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247335    2563 flags.go:64] FLAG: --seccomp-default="false"
Jul 29 14:34:58.255607 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247339    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 29 14:34:58.255607 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247342    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 29 14:34:58.255607 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247345    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 29 14:34:58.255607 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247347    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 29 14:34:58.255607 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247350    2563 flags.go:64] FLAG: --storage-driver-password="root"
Jul 29 14:34:58.255607 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247353    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 29 14:34:58.255607 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247356    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 29 14:34:58.255607 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247359    2563 flags.go:64] FLAG: --storage-driver-user="root"
Jul 29 14:34:58.255607 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247361    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 29 14:34:58.255607 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247364    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 29 14:34:58.255607 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247367    2563 flags.go:64] FLAG: --system-cgroups=""
Jul 29 14:34:58.255607 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247370    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 29 14:34:58.255607 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247375    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 29 14:34:58.255607 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247378    2563 flags.go:64] FLAG: --tls-cert-file=""
Jul 29 14:34:58.255607 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247380    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 29 14:34:58.255607 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247384    2563 flags.go:64] FLAG: --tls-min-version=""
Jul 29 14:34:58.255607 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247387    2563 flags.go:64] FLAG: --tls-private-key-file=""
Jul 29 14:34:58.255607 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247390    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 29 14:34:58.255607 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247393    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 29 14:34:58.255607 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247396    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 29 14:34:58.255607 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247399    2563 flags.go:64] FLAG: --v="2"
Jul 29 14:34:58.255607 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247403    2563 flags.go:64] FLAG: --version="false"
Jul 29 14:34:58.255607 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247407    2563 flags.go:64] FLAG: --vmodule=""
Jul 29 14:34:58.255607 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247412    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 29 14:34:58.256276 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.247415    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 29 14:34:58.256276 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247538    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 14:34:58.256276 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247542    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 14:34:58.256276 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247546    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 14:34:58.256276 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247549    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 14:34:58.256276 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247551    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 14:34:58.256276 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247554    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 14:34:58.256276 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247557    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 14:34:58.256276 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247559    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 14:34:58.256276 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247562    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 14:34:58.256276 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247564    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 14:34:58.256276 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247567    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 14:34:58.256276 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247569    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 14:34:58.256276 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247572    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 14:34:58.256276 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247575    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 14:34:58.256276 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247577    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 14:34:58.256276 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247580    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 14:34:58.256276 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247582    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 14:34:58.256276 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247584    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 14:34:58.256276 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247587    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 14:34:58.256786 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247590    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 14:34:58.256786 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247592    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 14:34:58.256786 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247594    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 14:34:58.256786 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247597    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 14:34:58.256786 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247599    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 14:34:58.256786 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247604    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 14:34:58.256786 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247606    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 14:34:58.256786 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247609    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 14:34:58.256786 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247611    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 14:34:58.256786 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247614    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 14:34:58.256786 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247616    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 14:34:58.256786 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247624    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 14:34:58.256786 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247627    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 14:34:58.256786 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247629    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 14:34:58.256786 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247632    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 14:34:58.256786 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247635    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 14:34:58.256786 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247637    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 14:34:58.256786 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247640    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 14:34:58.256786 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247643    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 14:34:58.256786 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247646    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 14:34:58.257304 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247648    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 14:34:58.257304 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247651    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 14:34:58.257304 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247653    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 14:34:58.257304 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247656    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 14:34:58.257304 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247658    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 14:34:58.257304 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247661    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 14:34:58.257304 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247664    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 14:34:58.257304 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247666    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 14:34:58.257304 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247669    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 14:34:58.257304 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247671    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 14:34:58.257304 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247674    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 14:34:58.257304 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247676    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 14:34:58.257304 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247678    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 14:34:58.257304 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247681    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 14:34:58.257304 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247685    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 14:34:58.257304 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247689    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 14:34:58.257304 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247692    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 14:34:58.257304 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247695    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 14:34:58.257304 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247698    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 14:34:58.257760 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247701    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 14:34:58.257760 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247703    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 14:34:58.257760 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247706    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 14:34:58.257760 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247709    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 14:34:58.257760 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247713    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 14:34:58.257760 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247716    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 14:34:58.257760 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247719    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 14:34:58.257760 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247721    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 14:34:58.257760 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247724    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 14:34:58.257760 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247726    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 14:34:58.257760 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247729    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 14:34:58.257760 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247732    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 14:34:58.257760 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247734    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 14:34:58.257760 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247737    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 14:34:58.257760 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247739    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 14:34:58.257760 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247742    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 14:34:58.257760 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247744    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 14:34:58.257760 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247747    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 14:34:58.257760 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247749    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 14:34:58.257760 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247752    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 14:34:58.258266 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247755    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 14:34:58.258266 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247757    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 14:34:58.258266 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247760    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 14:34:58.258266 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247762    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 14:34:58.258266 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247765    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 14:34:58.258266 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247767    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 14:34:58.258266 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247770    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 14:34:58.258266 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247772    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 14:34:58.258266 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247776    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 14:34:58.258266 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247786    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 14:34:58.258266 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247793    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 14:34:58.258266 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247796    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 14:34:58.258266 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247801    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 14:34:58.258266 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247804    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 14:34:58.258266 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247807    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 14:34:58.258266 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247810    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 14:34:58.258266 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247814    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 14:34:58.258266 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247817    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 14:34:58.258266 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247820    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 14:34:58.258724 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247823    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 14:34:58.258724 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.247826    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 14:34:58.258724 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.248551    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 14:34:58.258724 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.255700    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 29 14:34:58.258724 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.255717    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 29 14:34:58.258724 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.257155    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 29 14:34:58.258724 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.257177    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 29 14:34:58.258724 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257246    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 14:34:58.258724 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257253    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 14:34:58.258724 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257257    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 14:34:58.258724 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257261    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 14:34:58.258724 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257264    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 14:34:58.258724 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257266    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 14:34:58.258724 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257269    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 14:34:58.258724 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257272    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 14:34:58.259096 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257275    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 14:34:58.259096 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257278    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 14:34:58.259096 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257281    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 14:34:58.259096 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257284    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 14:34:58.259096 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257286    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 14:34:58.259096 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257289    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 14:34:58.259096 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257292    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 14:34:58.259096 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257295    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 14:34:58.259096 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257298    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 14:34:58.259096 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257301    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 14:34:58.259096 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257304    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 14:34:58.259096 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257307    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 14:34:58.259096 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257310    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 14:34:58.259096 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257313    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 14:34:58.259096 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257315    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 14:34:58.259096 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257318    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 14:34:58.259096 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257321    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 14:34:58.259096 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257323    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 14:34:58.259096 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257326    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 14:34:58.259096 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257328    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 14:34:58.259595 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257331    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 14:34:58.259595 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257337    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 14:34:58.259595 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257340    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 14:34:58.259595 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257343    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 14:34:58.259595 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257346    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 14:34:58.259595 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257349    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 14:34:58.259595 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257351    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 14:34:58.259595 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257354    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 14:34:58.259595 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257356    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 14:34:58.259595 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257359    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 14:34:58.259595 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257361    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 14:34:58.259595 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257364    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 14:34:58.259595 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257366    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 14:34:58.259595 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257369    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 14:34:58.259595 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257372    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 14:34:58.259595 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257374    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 14:34:58.259595 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257377    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 14:34:58.259595 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257380    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 14:34:58.259595 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257382    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 14:34:58.260061 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257387    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 14:34:58.260061 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257391    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 14:34:58.260061 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257394    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 14:34:58.260061 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257397    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 14:34:58.260061 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257399    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 14:34:58.260061 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257402    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 14:34:58.260061 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257406    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 14:34:58.260061 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257410    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 14:34:58.260061 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257414    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 14:34:58.260061 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257417    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 14:34:58.260061 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257420    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 14:34:58.260061 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257422    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 14:34:58.260061 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257425    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 14:34:58.260061 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257427    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 14:34:58.260061 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257430    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 14:34:58.260061 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257433    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 14:34:58.260061 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257436    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 14:34:58.260061 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257439    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 14:34:58.260061 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257442    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 14:34:58.260605 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257444    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 14:34:58.260605 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257447    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 14:34:58.260605 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257450    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 14:34:58.260605 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257452    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 14:34:58.260605 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257455    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 14:34:58.260605 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257458    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 14:34:58.260605 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257460    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 14:34:58.260605 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257463    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 14:34:58.260605 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257465    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 14:34:58.260605 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257468    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 14:34:58.260605 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257470    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 14:34:58.260605 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257473    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 14:34:58.260605 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257476    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 14:34:58.260605 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257478    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 14:34:58.260605 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257481    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 14:34:58.260605 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257483    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 14:34:58.260605 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257486    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 14:34:58.260605 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257489    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 14:34:58.260605 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257491    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 14:34:58.260605 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257494    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 14:34:58.261099 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257497    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 14:34:58.261099 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257500    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 14:34:58.261099 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257502    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 14:34:58.261099 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257505    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 14:34:58.261099 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257508    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 14:34:58.261099 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257510    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 14:34:58.261099 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257513    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 14:34:58.261099 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257515    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 14:34:58.261099 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257518    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 14:34:58.261099 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257521    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 14:34:58.261099 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257524    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 14:34:58.261099 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257526    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 14:34:58.261099 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257530    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 14:34:58.261099 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.257536    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 14:34:58.261099 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257647    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 14:34:58.261488 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257654    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 14:34:58.261488 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257657    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 14:34:58.261488 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257659    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 14:34:58.261488 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257662    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 14:34:58.261488 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257665    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 14:34:58.261488 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257667    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 14:34:58.261488 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257670    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 14:34:58.261488 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257672    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 14:34:58.261488 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257675    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 14:34:58.261488 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257677    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 14:34:58.261488 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257680    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 14:34:58.261488 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257682    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 14:34:58.261488 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257685    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 14:34:58.261488 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257687    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 14:34:58.261488 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257690    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 14:34:58.261488 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257692    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 14:34:58.261488 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257695    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 14:34:58.261488 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257697    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 14:34:58.261488 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257700    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 14:34:58.262009 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257702    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 14:34:58.262009 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257705    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 14:34:58.262009 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257707    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 14:34:58.262009 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257709    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 14:34:58.262009 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257712    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 14:34:58.262009 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257714    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 14:34:58.262009 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257717    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 14:34:58.262009 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257719    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 14:34:58.262009 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257723    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 14:34:58.262009 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257727    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 14:34:58.262009 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257731    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 14:34:58.262009 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257734    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 14:34:58.262009 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257736    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 14:34:58.262009 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257739    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 14:34:58.262009 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257741    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 14:34:58.262009 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257744    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 14:34:58.262009 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257746    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 14:34:58.262009 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257749    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 14:34:58.262009 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257751    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 14:34:58.262009 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257754    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 14:34:58.262509 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257756    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 14:34:58.262509 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257759    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 14:34:58.262509 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257762    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 14:34:58.262509 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257764    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 14:34:58.262509 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257767    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 14:34:58.262509 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257770    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 14:34:58.262509 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257772    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 14:34:58.262509 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257776    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 14:34:58.262509 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257779    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 14:34:58.262509 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257782    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 14:34:58.262509 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257784    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 14:34:58.262509 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257787    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 14:34:58.262509 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257790    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 14:34:58.262509 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257792    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 14:34:58.262509 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257794    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 14:34:58.262509 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257797    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 14:34:58.262509 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257799    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 14:34:58.262509 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257802    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 14:34:58.262509 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257805    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 14:34:58.262509 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257807    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 14:34:58.263069 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257810    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 14:34:58.263069 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257813    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 14:34:58.263069 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257816    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 14:34:58.263069 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257818    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 14:34:58.263069 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257821    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 14:34:58.263069 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257823    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 14:34:58.263069 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257826    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 14:34:58.263069 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257829    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 14:34:58.263069 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257831    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 14:34:58.263069 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257834    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 14:34:58.263069 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257836    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 14:34:58.263069 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257839    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 14:34:58.263069 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257841    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 14:34:58.263069 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257844    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 14:34:58.263069 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257846    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 14:34:58.263069 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257849    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 14:34:58.263069 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257851    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 14:34:58.263069 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257854    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 14:34:58.263069 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257857    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 14:34:58.263069 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257859    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 14:34:58.263667 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257862    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 14:34:58.263667 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257864    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 14:34:58.263667 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257867    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 14:34:58.263667 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257869    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 14:34:58.263667 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257872    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 14:34:58.263667 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257876    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 14:34:58.263667 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257879    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 14:34:58.263667 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257881    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 14:34:58.263667 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257883    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 14:34:58.263667 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257886    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 14:34:58.263667 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257889    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 14:34:58.263667 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257891    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 14:34:58.263667 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257894    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 14:34:58.263667 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257896    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 14:34:58.263667 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257899    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 14:34:58.263667 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257902    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 14:34:58.263667 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257904    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 14:34:58.263667 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257907    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 14:34:58.263667 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:58.257910    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 14:34:58.264308 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.257915    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 14:34:58.264308 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.258027    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 29 14:34:58.264308 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.260115    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 29 14:34:58.264308 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.261455    2563 server.go:1019] "Starting client certificate rotation"
Jul 29 14:34:58.264308 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.261574    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 14:34:58.264308 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.262640    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 14:34:58.264308 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.263773    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 14:34:58.264932 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.264909    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 14:34:58.276985 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.276964    2563 log.go:25] "Validated CRI v1 runtime API"
Jul 29 14:34:58.279181 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.279167    2563 log.go:25] "Validated CRI v1 image API"
Jul 29 14:34:58.280739 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.280717    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 29 14:34:58.283679 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.283653    2563 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 acbfeffd-2086-4c99-becc-afc51ed52595:/dev/nvme0n1p3 e0e46453-81c4-49c1-abcd-e0fd2f5602f7:/dev/nvme0n1p4]
Jul 29 14:34:58.283757 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.283677    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 29 14:34:58.288991 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.288876    2563 manager.go:216] Machine: {Timestamp:2026-07-29 14:34:58.28741157 +0000 UTC m=+0.335869500 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3090725 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e0de112bda50f4d85a78e618a4eb2 SystemUUID:ec2e0de1-12bd-a50f-4d85-a78e618a4eb2 BootID:6bf938e0-1959-4edd-b4bf-2b2638a1730e Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:80:26:a6:b3:85 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:80:26:a6:b3:85 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:3c:87:1e:5a:ae Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 29 14:34:58.288991 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.288980    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 29 14:34:58.289127 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.289061    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 29 14:34:58.290843 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.290818    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 29 14:34:58.291005 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.290848    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-196.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 29 14:34:58.291802 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.291790    2563 topology_manager.go:138] "Creating topology manager with none policy"
Jul 29 14:34:58.291858 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.291807    2563 container_manager_linux.go:309] "Creating device plugin manager"
Jul 29 14:34:58.291858 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.291819    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 14:34:58.291858 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.291835    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 14:34:58.292699 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.292687    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 29 14:34:58.294676 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.294657    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 14:34:58.294798 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.294785    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 14:34:58.294934 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.294923    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 29 14:34:58.294993 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.294962    2563 kubelet.go:486] "Attempting to sync node with API server"
Jul 29 14:34:58.294993 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.294978    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 29 14:34:58.295081 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.294994    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 29 14:34:58.295081 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.295009    2563 kubelet.go:398] "Adding apiserver pod source"
Jul 29 14:34:58.295081 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.295028    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 29 14:34:58.295437 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.295422    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 14:34:58.295499 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.295446    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 14:34:58.297836 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.297815    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 29 14:34:58.300108 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.300090    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 29 14:34:58.300188 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.300126    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 29 14:34:58.302062 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.302050    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 29 14:34:58.302109 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.302067    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 29 14:34:58.302109 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.302074    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 29 14:34:58.302109 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.302080    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 29 14:34:58.302109 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.302088    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 29 14:34:58.302109 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.302094    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 29 14:34:58.302109 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.302106    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 29 14:34:58.302109 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.302112    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 29 14:34:58.302308 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.302119    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 29 14:34:58.302308 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.302126    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 29 14:34:58.302308 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.302156    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 29 14:34:58.302308 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.302162    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 29 14:34:58.302968 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.302956    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 29 14:34:58.302968 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.302966    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 29 14:34:58.305358 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.305342    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-196.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 29 14:34:58.306024 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:34:58.305999    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-196.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 29 14:34:58.306024 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:34:58.306015    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 29 14:34:58.306189 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.306177    2563 server.go:1268] "Started kubelet"
Jul 29 14:34:58.306332 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.306303    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 29 14:34:58.306598 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.306287    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 29 14:34:58.306598 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.306505    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 29 14:34:58.306990 ip-10-0-136-196 systemd[1]: Started Kubernetes Kubelet.
Jul 29 14:34:58.310832 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.310805    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 29 14:34:58.312817 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.312518    2563 server.go:310] "Adding debug handlers to kubelet server"
Jul 29 14:34:58.315249 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:34:58.315232    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 29 14:34:58.318234 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:34:58.316211    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-196.ec2.internal.18c6c95bc65cb8c3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-196.ec2.internal,UID:ip-10-0-136-196.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-196.ec2.internal,},FirstTimestamp:2026-07-29 14:34:58.306152643 +0000 UTC m=+0.354610573,LastTimestamp:2026-07-29 14:34:58.306152643 +0000 UTC m=+0.354610573,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-196.ec2.internal,}"
Jul 29 14:34:58.320144 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.320114    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 29 14:34:58.320222 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.320155    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 29 14:34:58.320222 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:34:58.320207    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-196.ec2.internal\" not found"
Jul 29 14:34:58.320838 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.320825    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 29 14:34:58.320838 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.320826    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 29 14:34:58.320838 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.320844    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 29 14:34:58.320999 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.320951    2563 reconstruct.go:97] "Volume reconstruction finished"
Jul 29 14:34:58.320999 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.320959    2563 reconciler.go:29] "Reconciler: start to sync state"
Jul 29 14:34:58.321644 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.321628    2563 factory.go:55] Registering systemd factory
Jul 29 14:34:58.321644 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.321644    2563 factory.go:223] Registration of the systemd container factory successfully
Jul 29 14:34:58.321849 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.321837    2563 factory.go:153] Registering CRI-O factory
Jul 29 14:34:58.321849 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.321850    2563 factory.go:223] Registration of the crio container factory successfully
Jul 29 14:34:58.321976 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.321897    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 29 14:34:58.321976 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.321919    2563 factory.go:103] Registering Raw factory
Jul 29 14:34:58.321976 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.321934    2563 manager.go:1196] Started watching for new ooms in manager
Jul 29 14:34:58.322318 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.322305    2563 manager.go:349] Starting recovery of all containers
Jul 29 14:34:58.331399 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:34:58.331373    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-196.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 29 14:34:58.331399 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:34:58.331385    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 29 14:34:58.333220 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.333203    2563 manager.go:354] Recovery completed
Jul 29 14:34:58.337728 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.337717    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 14:34:58.339396 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.339376    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-196.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 14:34:58.339473 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.339408    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-196.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 14:34:58.339473 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.339433    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-196.ec2.internal" event="NodeHasSufficientPID"
Jul 29 14:34:58.340019 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.340005    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 29 14:34:58.340019 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.340017    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 29 14:34:58.340120 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.340032    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 14:34:58.340120 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.340008    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 14:34:58.340790 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.340774    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-196.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 14:34:58.340852 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.340801    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-196.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 14:34:58.340852 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.340811    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-196.ec2.internal" event="NodeHasSufficientPID"
Jul 29 14:34:58.341907 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.341894    2563 policy_none.go:49] "None policy: Start"
Jul 29 14:34:58.341945 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.341913    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 29 14:34:58.341945 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.341927    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 29 14:34:58.342967 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.342954    2563 policy_none.go:47] "Start"
Jul 29 14:34:58.344051 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.344037    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w9j74"
Jul 29 14:34:58.350302 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:34:58.350214    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-196.ec2.internal.18c6c95bc857fc0a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-196.ec2.internal,UID:ip-10-0-136-196.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-196.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-196.ec2.internal,},FirstTimestamp:2026-07-29 14:34:58.339396618 +0000 UTC m=+0.387854547,LastTimestamp:2026-07-29 14:34:58.339396618 +0000 UTC m=+0.387854547,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-196.ec2.internal,}"
Jul 29 14:34:58.352508 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.352485    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w9j74"
Jul 29 14:34:58.353425 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:34:58.353354    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-196.ec2.internal.18c6c95bc8583bcb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-196.ec2.internal,UID:ip-10-0-136-196.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-196.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-196.ec2.internal,},FirstTimestamp:2026-07-29 14:34:58.339412939 +0000 UTC m=+0.387870869,LastTimestamp:2026-07-29 14:34:58.339412939 +0000 UTC m=+0.387870869,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-196.ec2.internal,}"
Jul 29 14:34:58.372833 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.372816    2563 manager.go:337] "Starting Device Plugin manager"
Jul 29 14:34:58.389926 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:34:58.372859    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 29 14:34:58.389926 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.372871    2563 server.go:87] "Starting device plugin registration server"
Jul 29 14:34:58.389926 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.373107    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 29 14:34:58.389926 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.373116    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 29 14:34:58.389926 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.373311    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 29 14:34:58.389926 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.373388    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 29 14:34:58.389926 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.373396    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 29 14:34:58.389926 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:34:58.373790    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 29 14:34:58.389926 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:34:58.373839    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-196.ec2.internal\" not found"
Jul 29 14:34:58.424309 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.424273    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 29 14:34:58.425541 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.425514    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 29 14:34:58.425633 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.425554    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 29 14:34:58.425633 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.425578    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 29 14:34:58.425633 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:34:58.425617    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 29 14:34:58.428383 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.428367    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 14:34:58.473849 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.473820    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 14:34:58.474777 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.474757    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-196.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 14:34:58.474873 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.474792    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-196.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 14:34:58.474873 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.474806    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-196.ec2.internal" event="NodeHasSufficientPID"
Jul 29 14:34:58.474873 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.474835    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-196.ec2.internal"
Jul 29 14:34:58.486506 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.486489    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-196.ec2.internal"
Jul 29 14:34:58.486610 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:34:58.486514    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-196.ec2.internal\": node \"ip-10-0-136-196.ec2.internal\" not found"
Jul 29 14:34:58.518331 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:34:58.518263    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-196.ec2.internal\" not found"
Jul 29 14:34:58.526412 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.526386    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-196.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-196.ec2.internal"]
Jul 29 14:34:58.526513 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.526460    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 14:34:58.527457 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.527435    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-196.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 14:34:58.527530 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.527472    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-196.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 14:34:58.527530 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.527487    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-196.ec2.internal" event="NodeHasSufficientPID"
Jul 29 14:34:58.528780 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.528766    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 14:34:58.528950 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.528936    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-196.ec2.internal"
Jul 29 14:34:58.528987 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.528974    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 14:34:58.529507 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.529492    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-196.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 14:34:58.529579 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.529513    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-196.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 14:34:58.529579 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.529526    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-196.ec2.internal" event="NodeHasSufficientPID"
Jul 29 14:34:58.529579 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.529546    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-196.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 14:34:58.529579 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.529563    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-196.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 14:34:58.529579 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.529573    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-196.ec2.internal" event="NodeHasSufficientPID"
Jul 29 14:34:58.530836 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.530824    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-196.ec2.internal"
Jul 29 14:34:58.530888 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.530849    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 14:34:58.531489 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.531471    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-196.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 14:34:58.531577 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.531498    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-196.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 14:34:58.531577 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.531512    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-196.ec2.internal" event="NodeHasSufficientPID"
Jul 29 14:34:58.564857 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:34:58.564837    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-196.ec2.internal\" not found" node="ip-10-0-136-196.ec2.internal"
Jul 29 14:34:58.569238 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:34:58.569224    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-196.ec2.internal\" not found" node="ip-10-0-136-196.ec2.internal"
Jul 29 14:34:58.618389 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:34:58.618365    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-196.ec2.internal\" not found"
Jul 29 14:34:58.623148 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.623115    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c88b7e35b1a8b1d19afb95df94d0ce5f-config\") pod \"kube-apiserver-proxy-ip-10-0-136-196.ec2.internal\" (UID: \"c88b7e35b1a8b1d19afb95df94d0ce5f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-196.ec2.internal"
Jul 29 14:34:58.623232 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.623163    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/69153ffb63befe8c682d3ac1dcc08c24-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-196.ec2.internal\" (UID: \"69153ffb63befe8c682d3ac1dcc08c24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-196.ec2.internal"
Jul 29 14:34:58.623232 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.623191    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69153ffb63befe8c682d3ac1dcc08c24-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-196.ec2.internal\" (UID: \"69153ffb63befe8c682d3ac1dcc08c24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-196.ec2.internal"
Jul 29 14:34:58.719271 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:34:58.719238    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-196.ec2.internal\" not found"
Jul 29 14:34:58.723629 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.723602    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69153ffb63befe8c682d3ac1dcc08c24-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-196.ec2.internal\" (UID: \"69153ffb63befe8c682d3ac1dcc08c24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-196.ec2.internal"
Jul 29 14:34:58.723704 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.723638    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c88b7e35b1a8b1d19afb95df94d0ce5f-config\") pod \"kube-apiserver-proxy-ip-10-0-136-196.ec2.internal\" (UID: \"c88b7e35b1a8b1d19afb95df94d0ce5f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-196.ec2.internal"
Jul 29 14:34:58.723704 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.723659    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/69153ffb63befe8c682d3ac1dcc08c24-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-196.ec2.internal\" (UID: \"69153ffb63befe8c682d3ac1dcc08c24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-196.ec2.internal"
Jul 29 14:34:58.723704 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.723700    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/69153ffb63befe8c682d3ac1dcc08c24-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-196.ec2.internal\" (UID: \"69153ffb63befe8c682d3ac1dcc08c24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-196.ec2.internal"
Jul 29 14:34:58.723832 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.723714    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69153ffb63befe8c682d3ac1dcc08c24-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-196.ec2.internal\" (UID: \"69153ffb63befe8c682d3ac1dcc08c24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-196.ec2.internal"
Jul 29 14:34:58.723832 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.723719    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c88b7e35b1a8b1d19afb95df94d0ce5f-config\") pod \"kube-apiserver-proxy-ip-10-0-136-196.ec2.internal\" (UID: \"c88b7e35b1a8b1d19afb95df94d0ce5f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-196.ec2.internal"
Jul 29 14:34:58.820092 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:34:58.820004    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-196.ec2.internal\" not found"
Jul 29 14:34:58.867451 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.867421    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-196.ec2.internal"
Jul 29 14:34:58.872386 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:58.872367    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-196.ec2.internal"
Jul 29 14:34:58.920569 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:34:58.920536    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-196.ec2.internal\" not found"
Jul 29 14:34:59.021003 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:34:59.020957    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-196.ec2.internal\" not found"
Jul 29 14:34:59.121316 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:34:59.121245    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-196.ec2.internal\" not found"
Jul 29 14:34:59.227713 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:34:59.227672    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-196.ec2.internal\" not found"
Jul 29 14:34:59.260964 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:59.260937    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 29 14:34:59.261378 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:59.261114    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 14:34:59.328483 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:34:59.328453    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-196.ec2.internal\" not found"
Jul 29 14:34:59.354566 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:59.354536    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 14:34:59.355605 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:59.355588    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-196.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 14:34:59.355680 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:59.355626    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-196.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 14:34:59.355680 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:59.355642    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-196.ec2.internal" event="NodeHasSufficientPID"
Jul 29 14:34:59.358084 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:59.358049    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 14:29:58 +0000 UTC" deadline="2028-02-16 18:08:45.63899194 +0000 UTC"
Jul 29 14:34:59.358130 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:59.358085    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13611h33m46.280910201s"
Jul 29 14:34:59.381875 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:59.381823    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 14:34:59.428746 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:34:59.428722    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-196.ec2.internal\" not found"
Jul 29 14:34:59.529609 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:34:59.529576    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-196.ec2.internal\" not found"
Jul 29 14:34:59.630436 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:34:59.630390    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-196.ec2.internal\" not found"
Jul 29 14:34:59.715350 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:59.715269    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 14:34:59.721164 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:59.721129    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-196.ec2.internal"
Jul 29 14:34:59.732477 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:59.732458    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 14:34:59.734785 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:59.734769    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-196.ec2.internal"
Jul 29 14:34:59.746956 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:59.746938    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 14:34:59.868058 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:34:59.868025    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69153ffb63befe8c682d3ac1dcc08c24.slice/crio-8e93d4834d6bf947151b5bf2119c7fa4a29c6fbe7dba5e62cf230f5dc7050922 WatchSource:0}: Error finding container 8e93d4834d6bf947151b5bf2119c7fa4a29c6fbe7dba5e62cf230f5dc7050922: Status 404 returned error can't find the container with id 8e93d4834d6bf947151b5bf2119c7fa4a29c6fbe7dba5e62cf230f5dc7050922
Jul 29 14:34:59.873973 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:59.873951    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 14:34:59.908311 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:34:59.908283    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 14:35:00.040578 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:35:00.040549    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc88b7e35b1a8b1d19afb95df94d0ce5f.slice/crio-c61a63b84f8669c022f00128b0d53e82027cd61732b4e5cd5de3e676f15ed45d WatchSource:0}: Error finding container c61a63b84f8669c022f00128b0d53e82027cd61732b4e5cd5de3e676f15ed45d: Status 404 returned error can't find the container with id c61a63b84f8669c022f00128b0d53e82027cd61732b4e5cd5de3e676f15ed45d
Jul 29 14:35:00.297527 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.297424    2563 apiserver.go:52] "Watching apiserver"
Jul 29 14:35:00.302815 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.302792    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 29 14:35:00.304033 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.304009    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-d44hd","openshift-multus/multus-additional-cni-plugins-nwj5f","openshift-multus/multus-jsfjb","openshift-multus/network-metrics-daemon-vslsw","kube-system/global-pull-secret-syncer-vg8f5","openshift-dns/node-resolver-n7qwt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-196.ec2.internal","openshift-network-diagnostics/network-check-target-zvbtr","openshift-network-operator/iptables-alerter-l9xjd","openshift-ovn-kubernetes/ovnkube-node-8z2d5","kube-system/konnectivity-agent-k2jpl","kube-system/kube-apiserver-proxy-ip-10-0-136-196.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rzldt","openshift-cluster-node-tuning-operator/tuned-bh4w5"]
Jul 29 14:35:00.305767 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.305751    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d44hd"
Jul 29 14:35:00.306984 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.306958    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nwj5f"
Jul 29 14:35:00.307726 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.307701    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 29 14:35:00.307832 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.307701    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 29 14:35:00.307832 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.307783    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 29 14:35:00.308092 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.308068    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-j7sdl\""
Jul 29 14:35:00.308303 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.308288    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.309194 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.309178    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 29 14:35:00.309332 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.309223    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 29 14:35:00.309600 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.309570    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tqtvj\""
Jul 29 14:35:00.309721 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.309678    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 29 14:35:00.309721 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.309700    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 29 14:35:00.309858 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.309778    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 29 14:35:00.310027 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.310011    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.310154 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.310129    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 29 14:35:00.310258 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.310165    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-56l7r\""
Jul 29 14:35:00.311443 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.311425    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vg8f5"
Jul 29 14:35:00.311540 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:00.311478    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vg8f5" podUID="95701def-c5e7-4309-a07b-6a7c05316dd6"
Jul 29 14:35:00.311821 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.311804    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 29 14:35:00.312270 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.312234    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gx9l8\""
Jul 29 14:35:00.312270 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.312241    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 29 14:35:00.312369 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.312292    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 29 14:35:00.312369 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.312326    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 29 14:35:00.312727 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.312711    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 29 14:35:00.312788 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.312712    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 29 14:35:00.313075 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.313062    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n7qwt"
Jul 29 14:35:00.314767 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.314749    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvbtr"
Jul 29 14:35:00.314855 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:00.314811    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvbtr" podUID="e0828b12-3cef-4e42-8465-bf3dee289acb"
Jul 29 14:35:00.314946 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.314912    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 29 14:35:00.315407 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.315391    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fv8lx\""
Jul 29 14:35:00.315499 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.315482    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 29 14:35:00.316510 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.316495    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l9xjd"
Jul 29 14:35:00.319882 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.318397    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4q7j7\""
Jul 29 14:35:00.319882 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.318901    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 29 14:35:00.319882 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.319011    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 29 14:35:00.319882 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.319089    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 29 14:35:00.319882 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.319196    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vslsw"
Jul 29 14:35:00.319882 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:00.319276    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vslsw" podUID="ad392a5c-af96-47d5-acdd-9905fc1cf0d5"
Jul 29 14:35:00.320929 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.320899    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k2jpl"
Jul 29 14:35:00.321420 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.321401    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 29 14:35:00.322716 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.322698    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 29 14:35:00.322867 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.322849    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 29 14:35:00.322937 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.322909    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-6wppp\""
Jul 29 14:35:00.322990 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.322946    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rzldt"
Jul 29 14:35:00.324720 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.324701    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 29 14:35:00.324935 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.324916    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 29 14:35:00.324935 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.324926    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 29 14:35:00.325072 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.325015    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.325177 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.325160    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-blp8h\""
Jul 29 14:35:00.327967 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.327949    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 29 14:35:00.328128 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.328113    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 29 14:35:00.328229 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.328207    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pnkp7\""
Jul 29 14:35:00.333757 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.333732    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e666f634-f60c-47fb-952c-7e18a781609b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nwj5f\" (UID: \"e666f634-f60c-47fb-952c-7e18a781609b\") " pod="openshift-multus/multus-additional-cni-plugins-nwj5f"
Jul 29 14:35:00.333858 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.333772    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-systemd-units\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.333858 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.333804    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz8g8\" (UniqueName: \"kubernetes.io/projected/ad392a5c-af96-47d5-acdd-9905fc1cf0d5-kube-api-access-pz8g8\") pod \"network-metrics-daemon-vslsw\" (UID: \"ad392a5c-af96-47d5-acdd-9905fc1cf0d5\") " pod="openshift-multus/network-metrics-daemon-vslsw"
Jul 29 14:35:00.333858 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.333832    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vdq5\" (UniqueName: \"kubernetes.io/projected/75a67018-8010-4684-ac1c-4a12c7f30bea-kube-api-access-4vdq5\") pod \"node-ca-d44hd\" (UID: \"75a67018-8010-4684-ac1c-4a12c7f30bea\") " pod="openshift-image-registry/node-ca-d44hd"
Jul 29 14:35:00.334015 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.333856    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e666f634-f60c-47fb-952c-7e18a781609b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nwj5f\" (UID: \"e666f634-f60c-47fb-952c-7e18a781609b\") " pod="openshift-multus/multus-additional-cni-plugins-nwj5f"
Jul 29 14:35:00.334015 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.333870    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 14:35:00.334015 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.333881    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-host-cni-bin\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.334015 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.333919    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/95701def-c5e7-4309-a07b-6a7c05316dd6-dbus\") pod \"global-pull-secret-syncer-vg8f5\" (UID: \"95701def-c5e7-4309-a07b-6a7c05316dd6\") " pod="kube-system/global-pull-secret-syncer-vg8f5"
Jul 29 14:35:00.334015 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.333942    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-host-run-k8s-cni-cncf-io\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.334015 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.333960    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-run-openvswitch\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.334015 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.333986    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-host-run-ovn-kubernetes\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.334015 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334009    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c3ea9712-89d4-4d38-9b76-74f7c387f537-ovnkube-config\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.334309 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334025    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-system-cni-dir\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.334309 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334040    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa-sys-fs\") pod \"aws-ebs-csi-driver-node-rzldt\" (UID: \"4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rzldt"
Jul 29 14:35:00.334309 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334082    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqbh2\" (UniqueName: \"kubernetes.io/projected/4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa-kube-api-access-qqbh2\") pod \"aws-ebs-csi-driver-node-rzldt\" (UID: \"4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rzldt"
Jul 29 14:35:00.334309 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334108    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-host-run-netns\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.334309 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334127    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-run-ovn\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.334309 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334169    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ede7f878-5cc1-457e-8849-8ed44e5945bb-cni-binary-copy\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.334309 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334186    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e666f634-f60c-47fb-952c-7e18a781609b-os-release\") pod \"multus-additional-cni-plugins-nwj5f\" (UID: \"e666f634-f60c-47fb-952c-7e18a781609b\") " pod="openshift-multus/multus-additional-cni-plugins-nwj5f"
Jul 29 14:35:00.334309 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334202    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e666f634-f60c-47fb-952c-7e18a781609b-cni-binary-copy\") pod \"multus-additional-cni-plugins-nwj5f\" (UID: \"e666f634-f60c-47fb-952c-7e18a781609b\") " pod="openshift-multus/multus-additional-cni-plugins-nwj5f"
Jul 29 14:35:00.334309 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334224    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-run-systemd\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.334309 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334247    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-log-socket\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.334309 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334261    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-host-slash\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.334309 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334277    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95701def-c5e7-4309-a07b-6a7c05316dd6-original-pull-secret\") pod \"global-pull-secret-syncer-vg8f5\" (UID: \"95701def-c5e7-4309-a07b-6a7c05316dd6\") " pod="kube-system/global-pull-secret-syncer-vg8f5"
Jul 29 14:35:00.334309 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334295    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-host-run-netns\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.334772 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334319    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc5fa7a4-7058-45a4-a2f6-e3a260855a25-host-slash\") pod \"iptables-alerter-l9xjd\" (UID: \"cc5fa7a4-7058-45a4-a2f6-e3a260855a25\") " pod="openshift-network-operator/iptables-alerter-l9xjd"
Jul 29 14:35:00.334772 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334347    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rzldt\" (UID: \"4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rzldt"
Jul 29 14:35:00.334772 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334373    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa-registration-dir\") pod \"aws-ebs-csi-driver-node-rzldt\" (UID: \"4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rzldt"
Jul 29 14:35:00.334772 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334395    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-host-var-lib-cni-bin\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.334772 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334418    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa-socket-dir\") pod \"aws-ebs-csi-driver-node-rzldt\" (UID: \"4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rzldt"
Jul 29 14:35:00.334772 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334432    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75a67018-8010-4684-ac1c-4a12c7f30bea-host\") pod \"node-ca-d44hd\" (UID: \"75a67018-8010-4684-ac1c-4a12c7f30bea\") " pod="openshift-image-registry/node-ca-d44hd"
Jul 29 14:35:00.334772 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334449    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-node-log\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.334772 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334478    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-host-cni-netd\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.334772 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334499    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ede7f878-5cc1-457e-8849-8ed44e5945bb-multus-daemon-config\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.334772 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334528    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a010eab-ffa4-4f5a-a535-ecae3e55cf16-tmp-dir\") pod \"node-resolver-n7qwt\" (UID: \"8a010eab-ffa4-4f5a-a535-ecae3e55cf16\") " pod="openshift-dns/node-resolver-n7qwt"
Jul 29 14:35:00.334772 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334578    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-host-var-lib-cni-multus\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.334772 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334593    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-host-kubelet\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.334772 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334608    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-multus-cni-dir\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.334772 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334623    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-cnibin\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.334772 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334639    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p5xt\" (UniqueName: \"kubernetes.io/projected/cc5fa7a4-7058-45a4-a2f6-e3a260855a25-kube-api-access-4p5xt\") pod \"iptables-alerter-l9xjd\" (UID: \"cc5fa7a4-7058-45a4-a2f6-e3a260855a25\") " pod="openshift-network-operator/iptables-alerter-l9xjd"
Jul 29 14:35:00.334772 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334671    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-multus-socket-dir-parent\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.334772 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334694    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-host-run-multus-certs\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.335553 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334713    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e666f634-f60c-47fb-952c-7e18a781609b-system-cni-dir\") pod \"multus-additional-cni-plugins-nwj5f\" (UID: \"e666f634-f60c-47fb-952c-7e18a781609b\") " pod="openshift-multus/multus-additional-cni-plugins-nwj5f"
Jul 29 14:35:00.335553 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334746    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-etc-openvswitch\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.335553 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334768    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/95701def-c5e7-4309-a07b-6a7c05316dd6-kubelet-config\") pod \"global-pull-secret-syncer-vg8f5\" (UID: \"95701def-c5e7-4309-a07b-6a7c05316dd6\") " pod="kube-system/global-pull-secret-syncer-vg8f5"
Jul 29 14:35:00.335553 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334796    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa-etc-selinux\") pod \"aws-ebs-csi-driver-node-rzldt\" (UID: \"4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rzldt"
Jul 29 14:35:00.335553 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334813    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/75a67018-8010-4684-ac1c-4a12c7f30bea-serviceca\") pod \"node-ca-d44hd\" (UID: \"75a67018-8010-4684-ac1c-4a12c7f30bea\") " pod="openshift-image-registry/node-ca-d44hd"
Jul 29 14:35:00.335553 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334828    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c3ea9712-89d4-4d38-9b76-74f7c387f537-env-overrides\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.335553 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334844    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wltn7\" (UniqueName: \"kubernetes.io/projected/e0828b12-3cef-4e42-8465-bf3dee289acb-kube-api-access-wltn7\") pod \"network-check-target-zvbtr\" (UID: \"e0828b12-3cef-4e42-8465-bf3dee289acb\") " pod="openshift-network-diagnostics/network-check-target-zvbtr"
Jul 29 14:35:00.335553 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334859    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9a100b51-c157-4a63-a6c5-9d92a3fee8a7-agent-certs\") pod \"konnectivity-agent-k2jpl\" (UID: \"9a100b51-c157-4a63-a6c5-9d92a3fee8a7\") " pod="kube-system/konnectivity-agent-k2jpl"
Jul 29 14:35:00.335553 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334894    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-os-release\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.335553 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334935    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-hostroot\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.335553 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.334968    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c3ea9712-89d4-4d38-9b76-74f7c387f537-ovnkube-script-lib\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.335553 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.335004    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzl2x\" (UniqueName: \"kubernetes.io/projected/8a010eab-ffa4-4f5a-a535-ecae3e55cf16-kube-api-access-bzl2x\") pod \"node-resolver-n7qwt\" (UID: \"8a010eab-ffa4-4f5a-a535-ecae3e55cf16\") " pod="openshift-dns/node-resolver-n7qwt"
Jul 29 14:35:00.335553 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.335021    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9a100b51-c157-4a63-a6c5-9d92a3fee8a7-konnectivity-ca\") pod \"konnectivity-agent-k2jpl\" (UID: \"9a100b51-c157-4a63-a6c5-9d92a3fee8a7\") " pod="kube-system/konnectivity-agent-k2jpl"
Jul 29 14:35:00.335553 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.335048    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cc5fa7a4-7058-45a4-a2f6-e3a260855a25-iptables-alerter-script\") pod \"iptables-alerter-l9xjd\" (UID: \"cc5fa7a4-7058-45a4-a2f6-e3a260855a25\") " pod="openshift-network-operator/iptables-alerter-l9xjd"
Jul 29 14:35:00.335553 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.335064    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rp8l\" (UniqueName: \"kubernetes.io/projected/e666f634-f60c-47fb-952c-7e18a781609b-kube-api-access-2rp8l\") pod \"multus-additional-cni-plugins-nwj5f\" (UID: \"e666f634-f60c-47fb-952c-7e18a781609b\") " pod="openshift-multus/multus-additional-cni-plugins-nwj5f"
Jul 29 14:35:00.335553 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.335079    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-var-lib-openvswitch\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.336034 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.335093    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8a010eab-ffa4-4f5a-a535-ecae3e55cf16-hosts-file\") pod \"node-resolver-n7qwt\" (UID: \"8a010eab-ffa4-4f5a-a535-ecae3e55cf16\") " pod="openshift-dns/node-resolver-n7qwt"
Jul 29 14:35:00.336034 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.335107    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj6fr\" (UniqueName: \"kubernetes.io/projected/ede7f878-5cc1-457e-8849-8ed44e5945bb-kube-api-access-cj6fr\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.336034 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.335165    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa-device-dir\") pod \"aws-ebs-csi-driver-node-rzldt\" (UID: \"4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rzldt"
Jul 29 14:35:00.336034 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.335203    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.336034 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.335236    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c3ea9712-89d4-4d38-9b76-74f7c387f537-ovn-node-metrics-cert\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.336034 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.335265    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7ckx\" (UniqueName: \"kubernetes.io/projected/c3ea9712-89d4-4d38-9b76-74f7c387f537-kube-api-access-j7ckx\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.336034 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.335289    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-host-var-lib-kubelet\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.336034 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.335334    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-multus-conf-dir\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.336034 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.335376    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-etc-kubernetes\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.336034 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.335408    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad392a5c-af96-47d5-acdd-9905fc1cf0d5-metrics-certs\") pod \"network-metrics-daemon-vslsw\" (UID: \"ad392a5c-af96-47d5-acdd-9905fc1cf0d5\") " pod="openshift-multus/network-metrics-daemon-vslsw"
Jul 29 14:35:00.336034 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.335441    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e666f634-f60c-47fb-952c-7e18a781609b-cnibin\") pod \"multus-additional-cni-plugins-nwj5f\" (UID: \"e666f634-f60c-47fb-952c-7e18a781609b\") " pod="openshift-multus/multus-additional-cni-plugins-nwj5f"
Jul 29 14:35:00.336034 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.335466    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e666f634-f60c-47fb-952c-7e18a781609b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nwj5f\" (UID: \"e666f634-f60c-47fb-952c-7e18a781609b\") " pod="openshift-multus/multus-additional-cni-plugins-nwj5f"
Jul 29 14:35:00.360374 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.360347    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7qb5r"
Jul 29 14:35:00.371403 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.371384    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7qb5r"
Jul 29 14:35:00.422462 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.422428    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 29 14:35:00.430216 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.430137    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-196.ec2.internal" event={"ID":"c88b7e35b1a8b1d19afb95df94d0ce5f","Type":"ContainerStarted","Data":"c61a63b84f8669c022f00128b0d53e82027cd61732b4e5cd5de3e676f15ed45d"}
Jul 29 14:35:00.431796 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.431762    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-196.ec2.internal" event={"ID":"69153ffb63befe8c682d3ac1dcc08c24","Type":"ContainerStarted","Data":"8e93d4834d6bf947151b5bf2119c7fa4a29c6fbe7dba5e62cf230f5dc7050922"}
Jul 29 14:35:00.435937 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.435911    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-os-release\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.436070 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.435943    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-hostroot\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.436070 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.435966    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c3ea9712-89d4-4d38-9b76-74f7c387f537-ovnkube-script-lib\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.436070 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.435986    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzl2x\" (UniqueName: \"kubernetes.io/projected/8a010eab-ffa4-4f5a-a535-ecae3e55cf16-kube-api-access-bzl2x\") pod \"node-resolver-n7qwt\" (UID: \"8a010eab-ffa4-4f5a-a535-ecae3e55cf16\") " pod="openshift-dns/node-resolver-n7qwt"
Jul 29 14:35:00.436070 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.436010    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9a100b51-c157-4a63-a6c5-9d92a3fee8a7-konnectivity-ca\") pod \"konnectivity-agent-k2jpl\" (UID: \"9a100b51-c157-4a63-a6c5-9d92a3fee8a7\") " pod="kube-system/konnectivity-agent-k2jpl"
Jul 29 14:35:00.436070 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.436035    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cc5fa7a4-7058-45a4-a2f6-e3a260855a25-iptables-alerter-script\") pod \"iptables-alerter-l9xjd\" (UID: \"cc5fa7a4-7058-45a4-a2f6-e3a260855a25\") " pod="openshift-network-operator/iptables-alerter-l9xjd"
Jul 29 14:35:00.436070 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.436061    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rp8l\" (UniqueName: \"kubernetes.io/projected/e666f634-f60c-47fb-952c-7e18a781609b-kube-api-access-2rp8l\") pod \"multus-additional-cni-plugins-nwj5f\" (UID: \"e666f634-f60c-47fb-952c-7e18a781609b\") " pod="openshift-multus/multus-additional-cni-plugins-nwj5f"
Jul 29 14:35:00.436427 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.436062    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-hostroot\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.436427 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.436088    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-var-lib-openvswitch\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.436427 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.436117    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-etc-kubernetes\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.436427 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.436145    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-etc-systemd\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.436427 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.436172    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8a010eab-ffa4-4f5a-a535-ecae3e55cf16-hosts-file\") pod \"node-resolver-n7qwt\" (UID: \"8a010eab-ffa4-4f5a-a535-ecae3e55cf16\") " pod="openshift-dns/node-resolver-n7qwt"
Jul 29 14:35:00.436427 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.436253    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8a010eab-ffa4-4f5a-a535-ecae3e55cf16-hosts-file\") pod \"node-resolver-n7qwt\" (UID: \"8a010eab-ffa4-4f5a-a535-ecae3e55cf16\") " pod="openshift-dns/node-resolver-n7qwt"
Jul 29 14:35:00.436427 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.436324    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-var-lib-openvswitch\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.436427 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.436339    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj6fr\" (UniqueName: \"kubernetes.io/projected/ede7f878-5cc1-457e-8849-8ed44e5945bb-kube-api-access-cj6fr\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.436427 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.436368    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa-device-dir\") pod \"aws-ebs-csi-driver-node-rzldt\" (UID: \"4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rzldt"
Jul 29 14:35:00.436427 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.436396    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.436427 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.436423    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c3ea9712-89d4-4d38-9b76-74f7c387f537-ovn-node-metrics-cert\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.437013 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.436448    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7ckx\" (UniqueName: \"kubernetes.io/projected/c3ea9712-89d4-4d38-9b76-74f7c387f537-kube-api-access-j7ckx\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.437013 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.436467    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa-device-dir\") pod \"aws-ebs-csi-driver-node-rzldt\" (UID: \"4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rzldt"
Jul 29 14:35:00.437013 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.436477    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-etc-modprobe-d\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.437013 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.436503    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-lib-modules\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.437013 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.436531    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-host-var-lib-kubelet\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.437013 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.436558    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-multus-conf-dir\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.437013 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.436583    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-etc-kubernetes\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.437013 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.436610    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad392a5c-af96-47d5-acdd-9905fc1cf0d5-metrics-certs\") pod \"network-metrics-daemon-vslsw\" (UID: \"ad392a5c-af96-47d5-acdd-9905fc1cf0d5\") " pod="openshift-multus/network-metrics-daemon-vslsw"
Jul 29 14:35:00.437013 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.436637    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e666f634-f60c-47fb-952c-7e18a781609b-cnibin\") pod \"multus-additional-cni-plugins-nwj5f\" (UID: \"e666f634-f60c-47fb-952c-7e18a781609b\") " pod="openshift-multus/multus-additional-cni-plugins-nwj5f"
Jul 29 14:35:00.437013 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.436671    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e666f634-f60c-47fb-952c-7e18a781609b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nwj5f\" (UID: \"e666f634-f60c-47fb-952c-7e18a781609b\") " pod="openshift-multus/multus-additional-cni-plugins-nwj5f"
Jul 29 14:35:00.437013 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.436722    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e666f634-f60c-47fb-952c-7e18a781609b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nwj5f\" (UID: \"e666f634-f60c-47fb-952c-7e18a781609b\") " pod="openshift-multus/multus-additional-cni-plugins-nwj5f"
Jul 29 14:35:00.437013 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.436742    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c3ea9712-89d4-4d38-9b76-74f7c387f537-ovnkube-script-lib\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.437013 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.436717    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9a100b51-c157-4a63-a6c5-9d92a3fee8a7-konnectivity-ca\") pod \"konnectivity-agent-k2jpl\" (UID: \"9a100b51-c157-4a63-a6c5-9d92a3fee8a7\") " pod="kube-system/konnectivity-agent-k2jpl"
Jul 29 14:35:00.437013 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.436763    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 29 14:35:00.437013 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.436772    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.437013 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.436806    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-systemd-units\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.437013 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.436834    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-etc-kubernetes\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.437875 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.436841    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-os-release\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.437875 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.436860    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz8g8\" (UniqueName: \"kubernetes.io/projected/ad392a5c-af96-47d5-acdd-9905fc1cf0d5-kube-api-access-pz8g8\") pod \"network-metrics-daemon-vslsw\" (UID: \"ad392a5c-af96-47d5-acdd-9905fc1cf0d5\") " pod="openshift-multus/network-metrics-daemon-vslsw"
Jul 29 14:35:00.437875 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.436883    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-host-var-lib-kubelet\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.437875 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.436896    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vdq5\" (UniqueName: \"kubernetes.io/projected/75a67018-8010-4684-ac1c-4a12c7f30bea-kube-api-access-4vdq5\") pod \"node-ca-d44hd\" (UID: \"75a67018-8010-4684-ac1c-4a12c7f30bea\") " pod="openshift-image-registry/node-ca-d44hd"
Jul 29 14:35:00.437875 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.436914    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-multus-conf-dir\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.437875 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.436950    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e666f634-f60c-47fb-952c-7e18a781609b-cnibin\") pod \"multus-additional-cni-plugins-nwj5f\" (UID: \"e666f634-f60c-47fb-952c-7e18a781609b\") " pod="openshift-multus/multus-additional-cni-plugins-nwj5f"
Jul 29 14:35:00.437875 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.436959    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-systemd-units\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.437875 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.436981    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e666f634-f60c-47fb-952c-7e18a781609b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nwj5f\" (UID: \"e666f634-f60c-47fb-952c-7e18a781609b\") " pod="openshift-multus/multus-additional-cni-plugins-nwj5f"
Jul 29 14:35:00.437875 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437008    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-host-cni-bin\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.437875 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437032    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/95701def-c5e7-4309-a07b-6a7c05316dd6-dbus\") pod \"global-pull-secret-syncer-vg8f5\" (UID: \"95701def-c5e7-4309-a07b-6a7c05316dd6\") " pod="kube-system/global-pull-secret-syncer-vg8f5"
Jul 29 14:35:00.437875 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:00.437050    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 14:35:00.437875 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437057    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-host-run-k8s-cni-cncf-io\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.437875 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437084    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-run-openvswitch\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.437875 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437108    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-host-run-ovn-kubernetes\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.437875 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:00.437127    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad392a5c-af96-47d5-acdd-9905fc1cf0d5-metrics-certs podName:ad392a5c-af96-47d5-acdd-9905fc1cf0d5 nodeName:}" failed. No retries permitted until 2026-07-29 14:35:00.937104676 +0000 UTC m=+2.985562617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ad392a5c-af96-47d5-acdd-9905fc1cf0d5-metrics-certs") pod "network-metrics-daemon-vslsw" (UID: "ad392a5c-af96-47d5-acdd-9905fc1cf0d5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 14:35:00.437875 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437159    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c3ea9712-89d4-4d38-9b76-74f7c387f537-ovnkube-config\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.437875 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437180    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-host-run-ovn-kubernetes\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.438719 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437189    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-host\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.438719 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437221    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e666f634-f60c-47fb-952c-7e18a781609b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nwj5f\" (UID: \"e666f634-f60c-47fb-952c-7e18a781609b\") " pod="openshift-multus/multus-additional-cni-plugins-nwj5f"
Jul 29 14:35:00.438719 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437236    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-system-cni-dir\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.438719 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437264    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa-sys-fs\") pod \"aws-ebs-csi-driver-node-rzldt\" (UID: \"4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rzldt"
Jul 29 14:35:00.438719 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437294    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqbh2\" (UniqueName: \"kubernetes.io/projected/4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa-kube-api-access-qqbh2\") pod \"aws-ebs-csi-driver-node-rzldt\" (UID: \"4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rzldt"
Jul 29 14:35:00.438719 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437321    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e666f634-f60c-47fb-952c-7e18a781609b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nwj5f\" (UID: \"e666f634-f60c-47fb-952c-7e18a781609b\") " pod="openshift-multus/multus-additional-cni-plugins-nwj5f"
Jul 29 14:35:00.438719 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437335    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-run-openvswitch\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.438719 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437337    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-system-cni-dir\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.438719 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437327    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-host-run-netns\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.438719 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437364    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-host-run-netns\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.438719 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437237    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-host-cni-bin\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.438719 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437390    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-run-ovn\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.438719 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437419    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-etc-sysconfig\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.438719 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437445    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ede7f878-5cc1-457e-8849-8ed44e5945bb-cni-binary-copy\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.438719 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437462    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa-sys-fs\") pod \"aws-ebs-csi-driver-node-rzldt\" (UID: \"4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rzldt"
Jul 29 14:35:00.438719 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437267    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/95701def-c5e7-4309-a07b-6a7c05316dd6-dbus\") pod \"global-pull-secret-syncer-vg8f5\" (UID: \"95701def-c5e7-4309-a07b-6a7c05316dd6\") " pod="kube-system/global-pull-secret-syncer-vg8f5"
Jul 29 14:35:00.438719 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437511    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e666f634-f60c-47fb-952c-7e18a781609b-os-release\") pod \"multus-additional-cni-plugins-nwj5f\" (UID: \"e666f634-f60c-47fb-952c-7e18a781609b\") " pod="openshift-multus/multus-additional-cni-plugins-nwj5f"
Jul 29 14:35:00.439528 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437543    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e666f634-f60c-47fb-952c-7e18a781609b-cni-binary-copy\") pod \"multus-additional-cni-plugins-nwj5f\" (UID: \"e666f634-f60c-47fb-952c-7e18a781609b\") " pod="openshift-multus/multus-additional-cni-plugins-nwj5f"
Jul 29 14:35:00.439528 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437569    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-run-systemd\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.439528 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437594    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-log-socket\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.439528 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437622    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-run\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.439528 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437649    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-host-slash\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.439528 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437675    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95701def-c5e7-4309-a07b-6a7c05316dd6-original-pull-secret\") pod \"global-pull-secret-syncer-vg8f5\" (UID: \"95701def-c5e7-4309-a07b-6a7c05316dd6\") " pod="kube-system/global-pull-secret-syncer-vg8f5"
Jul 29 14:35:00.439528 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437723    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-host-run-netns\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.439528 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437755    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc5fa7a4-7058-45a4-a2f6-e3a260855a25-host-slash\") pod \"iptables-alerter-l9xjd\" (UID: \"cc5fa7a4-7058-45a4-a2f6-e3a260855a25\") " pod="openshift-network-operator/iptables-alerter-l9xjd"
Jul 29 14:35:00.439528 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437781    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rzldt\" (UID: \"4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rzldt"
Jul 29 14:35:00.439528 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437791    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e666f634-f60c-47fb-952c-7e18a781609b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nwj5f\" (UID: \"e666f634-f60c-47fb-952c-7e18a781609b\") " pod="openshift-multus/multus-additional-cni-plugins-nwj5f"
Jul 29 14:35:00.439528 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437801    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c3ea9712-89d4-4d38-9b76-74f7c387f537-ovnkube-config\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.439528 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437810    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa-registration-dir\") pod \"aws-ebs-csi-driver-node-rzldt\" (UID: \"4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rzldt"
Jul 29 14:35:00.439528 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437298    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-host-run-k8s-cni-cncf-io\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.439528 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437838    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-host-var-lib-cni-bin\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.439528 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437864    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa-socket-dir\") pod \"aws-ebs-csi-driver-node-rzldt\" (UID: \"4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rzldt"
Jul 29 14:35:00.439528 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437892    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75a67018-8010-4684-ac1c-4a12c7f30bea-host\") pod \"node-ca-d44hd\" (UID: \"75a67018-8010-4684-ac1c-4a12c7f30bea\") " pod="openshift-image-registry/node-ca-d44hd"
Jul 29 14:35:00.439528 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437916    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-node-log\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.440282 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437932    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-host-cni-netd\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.440282 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437957    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ede7f878-5cc1-457e-8849-8ed44e5945bb-multus-daemon-config\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.440282 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437957    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ede7f878-5cc1-457e-8849-8ed44e5945bb-cni-binary-copy\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.440282 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437966    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-run-ovn\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.440282 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437976    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a010eab-ffa4-4f5a-a535-ecae3e55cf16-tmp-dir\") pod \"node-resolver-n7qwt\" (UID: \"8a010eab-ffa4-4f5a-a535-ecae3e55cf16\") " pod="openshift-dns/node-resolver-n7qwt"
Jul 29 14:35:00.440282 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.437996    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-host-var-lib-cni-multus\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.440282 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438012    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-host-kubelet\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.440282 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438016    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc5fa7a4-7058-45a4-a2f6-e3a260855a25-host-slash\") pod \"iptables-alerter-l9xjd\" (UID: \"cc5fa7a4-7058-45a4-a2f6-e3a260855a25\") " pod="openshift-network-operator/iptables-alerter-l9xjd"
Jul 29 14:35:00.440282 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438024    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-log-socket\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.440282 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438031    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-tmp\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.440282 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438059    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75a67018-8010-4684-ac1c-4a12c7f30bea-host\") pod \"node-ca-d44hd\" (UID: \"75a67018-8010-4684-ac1c-4a12c7f30bea\") " pod="openshift-image-registry/node-ca-d44hd"
Jul 29 14:35:00.440282 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438075    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7kcn\" (UniqueName: \"kubernetes.io/projected/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-kube-api-access-p7kcn\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.440282 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438086    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-run-systemd\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.440282 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438103    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-multus-cni-dir\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.440282 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438129    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-cnibin\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.440282 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438135    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-host-var-lib-cni-bin\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.440282 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:00.438149    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 14:35:00.440282 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438154    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p5xt\" (UniqueName: \"kubernetes.io/projected/cc5fa7a4-7058-45a4-a2f6-e3a260855a25-kube-api-access-4p5xt\") pod \"iptables-alerter-l9xjd\" (UID: \"cc5fa7a4-7058-45a4-a2f6-e3a260855a25\") " pod="openshift-network-operator/iptables-alerter-l9xjd"
Jul 29 14:35:00.440996 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438173    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-etc-sysctl-conf\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.440996 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438193    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-multus-socket-dir-parent\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.440996 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438193    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-node-log\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.440996 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:00.438224    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95701def-c5e7-4309-a07b-6a7c05316dd6-original-pull-secret podName:95701def-c5e7-4309-a07b-6a7c05316dd6 nodeName:}" failed. No retries permitted until 2026-07-29 14:35:00.938189302 +0000 UTC m=+2.986647224 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/95701def-c5e7-4309-a07b-6a7c05316dd6-original-pull-secret") pod "global-pull-secret-syncer-vg8f5" (UID: "95701def-c5e7-4309-a07b-6a7c05316dd6") : object "kube-system"/"original-pull-secret" not registered
Jul 29 14:35:00.440996 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438251    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-host-cni-netd\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.440996 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438256    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa-socket-dir\") pod \"aws-ebs-csi-driver-node-rzldt\" (UID: \"4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rzldt"
Jul 29 14:35:00.440996 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438255    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-multus-socket-dir-parent\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.440996 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438266    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-host-run-netns\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.440996 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438276    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-host-slash\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.440996 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438327    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-host-run-multus-certs\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.440996 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438331    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e666f634-f60c-47fb-952c-7e18a781609b-os-release\") pod \"multus-additional-cni-plugins-nwj5f\" (UID: \"e666f634-f60c-47fb-952c-7e18a781609b\") " pod="openshift-multus/multus-additional-cni-plugins-nwj5f"
Jul 29 14:35:00.440996 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438337    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-multus-cni-dir\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.440996 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438358    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e666f634-f60c-47fb-952c-7e18a781609b-system-cni-dir\") pod \"multus-additional-cni-plugins-nwj5f\" (UID: \"e666f634-f60c-47fb-952c-7e18a781609b\") " pod="openshift-multus/multus-additional-cni-plugins-nwj5f"
Jul 29 14:35:00.440996 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438386    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-etc-openvswitch\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.440996 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438388    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa-registration-dir\") pod \"aws-ebs-csi-driver-node-rzldt\" (UID: \"4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rzldt"
Jul 29 14:35:00.440996 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438136    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rzldt\" (UID: \"4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rzldt"
Jul 29 14:35:00.441674 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438414    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-sys\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.441674 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438444    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-host-var-lib-cni-multus\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.441674 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438440    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-var-lib-kubelet\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.441674 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438493    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-etc-tuned\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.441674 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438504    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-host-kubelet\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.441674 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438520    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/95701def-c5e7-4309-a07b-6a7c05316dd6-kubelet-config\") pod \"global-pull-secret-syncer-vg8f5\" (UID: \"95701def-c5e7-4309-a07b-6a7c05316dd6\") " pod="kube-system/global-pull-secret-syncer-vg8f5"
Jul 29 14:35:00.441674 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438532    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-cnibin\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.441674 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438549    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa-etc-selinux\") pod \"aws-ebs-csi-driver-node-rzldt\" (UID: \"4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rzldt"
Jul 29 14:35:00.441674 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438575    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/75a67018-8010-4684-ac1c-4a12c7f30bea-serviceca\") pod \"node-ca-d44hd\" (UID: \"75a67018-8010-4684-ac1c-4a12c7f30bea\") " pod="openshift-image-registry/node-ca-d44hd"
Jul 29 14:35:00.441674 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438594    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ede7f878-5cc1-457e-8849-8ed44e5945bb-host-run-multus-certs\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.441674 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438603    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c3ea9712-89d4-4d38-9b76-74f7c387f537-env-overrides\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.441674 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438630    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-etc-sysctl-d\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.441674 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438637    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e666f634-f60c-47fb-952c-7e18a781609b-system-cni-dir\") pod \"multus-additional-cni-plugins-nwj5f\" (UID: \"e666f634-f60c-47fb-952c-7e18a781609b\") " pod="openshift-multus/multus-additional-cni-plugins-nwj5f"
Jul 29 14:35:00.441674 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438678    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wltn7\" (UniqueName: \"kubernetes.io/projected/e0828b12-3cef-4e42-8465-bf3dee289acb-kube-api-access-wltn7\") pod \"network-check-target-zvbtr\" (UID: \"e0828b12-3cef-4e42-8465-bf3dee289acb\") " pod="openshift-network-diagnostics/network-check-target-zvbtr"
Jul 29 14:35:00.441674 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438684    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3ea9712-89d4-4d38-9b76-74f7c387f537-etc-openvswitch\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.441674 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438549    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a010eab-ffa4-4f5a-a535-ecae3e55cf16-tmp-dir\") pod \"node-resolver-n7qwt\" (UID: \"8a010eab-ffa4-4f5a-a535-ecae3e55cf16\") " pod="openshift-dns/node-resolver-n7qwt"
Jul 29 14:35:00.441674 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438729    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9a100b51-c157-4a63-a6c5-9d92a3fee8a7-agent-certs\") pod \"konnectivity-agent-k2jpl\" (UID: \"9a100b51-c157-4a63-a6c5-9d92a3fee8a7\") " pod="kube-system/konnectivity-agent-k2jpl"
Jul 29 14:35:00.442316 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438737    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e666f634-f60c-47fb-952c-7e18a781609b-cni-binary-copy\") pod \"multus-additional-cni-plugins-nwj5f\" (UID: \"e666f634-f60c-47fb-952c-7e18a781609b\") " pod="openshift-multus/multus-additional-cni-plugins-nwj5f"
Jul 29 14:35:00.442316 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438737    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/95701def-c5e7-4309-a07b-6a7c05316dd6-kubelet-config\") pod \"global-pull-secret-syncer-vg8f5\" (UID: \"95701def-c5e7-4309-a07b-6a7c05316dd6\") " pod="kube-system/global-pull-secret-syncer-vg8f5"
Jul 29 14:35:00.442316 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.438907    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa-etc-selinux\") pod \"aws-ebs-csi-driver-node-rzldt\" (UID: \"4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rzldt"
Jul 29 14:35:00.442316 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.439054    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ede7f878-5cc1-457e-8849-8ed44e5945bb-multus-daemon-config\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.442316 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.439203    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c3ea9712-89d4-4d38-9b76-74f7c387f537-env-overrides\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.442316 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.439256    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/75a67018-8010-4684-ac1c-4a12c7f30bea-serviceca\") pod \"node-ca-d44hd\" (UID: \"75a67018-8010-4684-ac1c-4a12c7f30bea\") " pod="openshift-image-registry/node-ca-d44hd"
Jul 29 14:35:00.442316 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.439438    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cc5fa7a4-7058-45a4-a2f6-e3a260855a25-iptables-alerter-script\") pod \"iptables-alerter-l9xjd\" (UID: \"cc5fa7a4-7058-45a4-a2f6-e3a260855a25\") " pod="openshift-network-operator/iptables-alerter-l9xjd"
Jul 29 14:35:00.442316 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.440911    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c3ea9712-89d4-4d38-9b76-74f7c387f537-ovn-node-metrics-cert\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.442316 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.441499    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9a100b51-c157-4a63-a6c5-9d92a3fee8a7-agent-certs\") pod \"konnectivity-agent-k2jpl\" (UID: \"9a100b51-c157-4a63-a6c5-9d92a3fee8a7\") " pod="kube-system/konnectivity-agent-k2jpl"
Jul 29 14:35:00.445918 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.445843    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzl2x\" (UniqueName: \"kubernetes.io/projected/8a010eab-ffa4-4f5a-a535-ecae3e55cf16-kube-api-access-bzl2x\") pod \"node-resolver-n7qwt\" (UID: \"8a010eab-ffa4-4f5a-a535-ecae3e55cf16\") " pod="openshift-dns/node-resolver-n7qwt"
Jul 29 14:35:00.446071 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.446049    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj6fr\" (UniqueName: \"kubernetes.io/projected/ede7f878-5cc1-457e-8849-8ed44e5945bb-kube-api-access-cj6fr\") pod \"multus-jsfjb\" (UID: \"ede7f878-5cc1-457e-8849-8ed44e5945bb\") " pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.446515 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.446489    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rp8l\" (UniqueName: \"kubernetes.io/projected/e666f634-f60c-47fb-952c-7e18a781609b-kube-api-access-2rp8l\") pod \"multus-additional-cni-plugins-nwj5f\" (UID: \"e666f634-f60c-47fb-952c-7e18a781609b\") " pod="openshift-multus/multus-additional-cni-plugins-nwj5f"
Jul 29 14:35:00.447634 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:00.447312    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 14:35:00.447634 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:00.447332    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 14:35:00.447634 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:00.447346    2563 projected.go:196] Error preparing data for projected volume kube-api-access-wltn7 for pod openshift-network-diagnostics/network-check-target-zvbtr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 14:35:00.447634 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:00.447403    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0828b12-3cef-4e42-8465-bf3dee289acb-kube-api-access-wltn7 podName:e0828b12-3cef-4e42-8465-bf3dee289acb nodeName:}" failed. No retries permitted until 2026-07-29 14:35:00.947390245 +0000 UTC m=+2.995848164 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wltn7" (UniqueName: "kubernetes.io/projected/e0828b12-3cef-4e42-8465-bf3dee289acb-kube-api-access-wltn7") pod "network-check-target-zvbtr" (UID: "e0828b12-3cef-4e42-8465-bf3dee289acb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 14:35:00.447634 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.447590    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7ckx\" (UniqueName: \"kubernetes.io/projected/c3ea9712-89d4-4d38-9b76-74f7c387f537-kube-api-access-j7ckx\") pod \"ovnkube-node-8z2d5\" (UID: \"c3ea9712-89d4-4d38-9b76-74f7c387f537\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.447954 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.447866    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vdq5\" (UniqueName: \"kubernetes.io/projected/75a67018-8010-4684-ac1c-4a12c7f30bea-kube-api-access-4vdq5\") pod \"node-ca-d44hd\" (UID: \"75a67018-8010-4684-ac1c-4a12c7f30bea\") " pod="openshift-image-registry/node-ca-d44hd"
Jul 29 14:35:00.447954 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.447923    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqbh2\" (UniqueName: \"kubernetes.io/projected/4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa-kube-api-access-qqbh2\") pod \"aws-ebs-csi-driver-node-rzldt\" (UID: \"4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rzldt"
Jul 29 14:35:00.448342 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.448319    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz8g8\" (UniqueName: \"kubernetes.io/projected/ad392a5c-af96-47d5-acdd-9905fc1cf0d5-kube-api-access-pz8g8\") pod \"network-metrics-daemon-vslsw\" (UID: \"ad392a5c-af96-47d5-acdd-9905fc1cf0d5\") " pod="openshift-multus/network-metrics-daemon-vslsw"
Jul 29 14:35:00.448450 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.448421    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p5xt\" (UniqueName: \"kubernetes.io/projected/cc5fa7a4-7058-45a4-a2f6-e3a260855a25-kube-api-access-4p5xt\") pod \"iptables-alerter-l9xjd\" (UID: \"cc5fa7a4-7058-45a4-a2f6-e3a260855a25\") " pod="openshift-network-operator/iptables-alerter-l9xjd"
Jul 29 14:35:00.539270 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.539219    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-etc-modprobe-d\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.539270 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.539264    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-lib-modules\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.539525 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.539316    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-host\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.539525 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.539343    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-etc-sysconfig\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.539525 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.539367    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-run\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.539525 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.539396    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-etc-modprobe-d\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.539525 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.539417    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-lib-modules\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.539525 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.539404    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-tmp\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.539525 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.539458    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7kcn\" (UniqueName: \"kubernetes.io/projected/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-kube-api-access-p7kcn\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.539845 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.539424    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-host\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.539845 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.539486    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-etc-sysctl-conf\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.539845 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.539458    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-run\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.539992 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.539452    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-etc-sysconfig\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.539992 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.539982    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-sys\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.540095 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.540021    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-var-lib-kubelet\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.540095 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.540063    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-etc-tuned\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.540095 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.540079    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-etc-sysctl-conf\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.540286 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.540103    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-etc-sysctl-d\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.540286 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.540196    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-etc-kubernetes\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.540286 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.540201    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-sys\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.540432 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.540327    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-etc-kubernetes\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.540432 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.540378    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-var-lib-kubelet\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.540432 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.540412    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-etc-sysctl-d\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.540598 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.540462    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-etc-systemd\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.540598 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.540584    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-etc-systemd\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.543770 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.543138    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-tmp\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.545276 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.545253    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-etc-tuned\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.548124 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.548071    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7kcn\" (UniqueName: \"kubernetes.io/projected/8dfac3ca-72cd-4bf4-8309-3774c7abe57a-kube-api-access-p7kcn\") pod \"tuned-bh4w5\" (UID: \"8dfac3ca-72cd-4bf4-8309-3774c7abe57a\") " pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.619060 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.619021    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d44hd"
Jul 29 14:35:00.624725 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.624697    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nwj5f"
Jul 29 14:35:00.631705 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.631664    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jsfjb"
Jul 29 14:35:00.632353 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:35:00.632316    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75a67018_8010_4684_ac1c_4a12c7f30bea.slice/crio-60210ed630035a0e5406bb31bb40c5f75c4022d913091c181e3105e60eb5fbf6 WatchSource:0}: Error finding container 60210ed630035a0e5406bb31bb40c5f75c4022d913091c181e3105e60eb5fbf6: Status 404 returned error can't find the container with id 60210ed630035a0e5406bb31bb40c5f75c4022d913091c181e3105e60eb5fbf6
Jul 29 14:35:00.635572 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:35:00.635548    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode666f634_f60c_47fb_952c_7e18a781609b.slice/crio-aaa62b71ebab08bcfcba4423c36c9d79044ed81c56748900cd1d9f43adfa1911 WatchSource:0}: Error finding container aaa62b71ebab08bcfcba4423c36c9d79044ed81c56748900cd1d9f43adfa1911: Status 404 returned error can't find the container with id aaa62b71ebab08bcfcba4423c36c9d79044ed81c56748900cd1d9f43adfa1911
Jul 29 14:35:00.636342 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.636321    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:00.642034 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.642012    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n7qwt"
Jul 29 14:35:00.645021 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:35:00.644991    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede7f878_5cc1_457e_8849_8ed44e5945bb.slice/crio-e6520a73552332ca56e62aaeec5a4ea1d787736b2cbf01d9cb90a1f68de75766 WatchSource:0}: Error finding container e6520a73552332ca56e62aaeec5a4ea1d787736b2cbf01d9cb90a1f68de75766: Status 404 returned error can't find the container with id e6520a73552332ca56e62aaeec5a4ea1d787736b2cbf01d9cb90a1f68de75766
Jul 29 14:35:00.646229 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:35:00.646198    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3ea9712_89d4_4d38_9b76_74f7c387f537.slice/crio-08baa13cc74b5049b95683fae67094a4adf611e88131b6990e25f199848d0ee3 WatchSource:0}: Error finding container 08baa13cc74b5049b95683fae67094a4adf611e88131b6990e25f199848d0ee3: Status 404 returned error can't find the container with id 08baa13cc74b5049b95683fae67094a4adf611e88131b6990e25f199848d0ee3
Jul 29 14:35:00.648613 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.648551    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l9xjd"
Jul 29 14:35:00.652154 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:35:00.652110    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a010eab_ffa4_4f5a_a535_ecae3e55cf16.slice/crio-cf6bc0ef1427d8af39be02b2a0440785f8aaf259fbe0cec72d8e94c91b3803ea WatchSource:0}: Error finding container cf6bc0ef1427d8af39be02b2a0440785f8aaf259fbe0cec72d8e94c91b3803ea: Status 404 returned error can't find the container with id cf6bc0ef1427d8af39be02b2a0440785f8aaf259fbe0cec72d8e94c91b3803ea
Jul 29 14:35:00.655026 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.655008    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k2jpl"
Jul 29 14:35:00.659288 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:35:00.659258    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc5fa7a4_7058_45a4_a2f6_e3a260855a25.slice/crio-cee43bbbe60553d64d45f9f0031c762307f1fc26c1d335c53dbfc791dabad67c WatchSource:0}: Error finding container cee43bbbe60553d64d45f9f0031c762307f1fc26c1d335c53dbfc791dabad67c: Status 404 returned error can't find the container with id cee43bbbe60553d64d45f9f0031c762307f1fc26c1d335c53dbfc791dabad67c
Jul 29 14:35:00.660671 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.660650    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rzldt"
Jul 29 14:35:00.664806 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:35:00.664784    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a100b51_c157_4a63_a6c5_9d92a3fee8a7.slice/crio-bd4ab80e0cdd10597d08f783650c4ba3c5bac6f713ec763fa1bf775530e19c1b WatchSource:0}: Error finding container bd4ab80e0cdd10597d08f783650c4ba3c5bac6f713ec763fa1bf775530e19c1b: Status 404 returned error can't find the container with id bd4ab80e0cdd10597d08f783650c4ba3c5bac6f713ec763fa1bf775530e19c1b
Jul 29 14:35:00.665586 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.665527    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bh4w5"
Jul 29 14:35:00.671450 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:35:00.671406    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f2d7ae6_f34b_4d7e_be9a_587701d9c0aa.slice/crio-5be6bce154ad5c4aad3d4b1713c09716e7ac5ab206df6d5c3cda38e07ab75788 WatchSource:0}: Error finding container 5be6bce154ad5c4aad3d4b1713c09716e7ac5ab206df6d5c3cda38e07ab75788: Status 404 returned error can't find the container with id 5be6bce154ad5c4aad3d4b1713c09716e7ac5ab206df6d5c3cda38e07ab75788
Jul 29 14:35:00.676217 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:35:00.676180    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dfac3ca_72cd_4bf4_8309_3774c7abe57a.slice/crio-860d70284c8731717f847d760ad4cb1eafd2b58189e6aa542a283ea02ba3c9d5 WatchSource:0}: Error finding container 860d70284c8731717f847d760ad4cb1eafd2b58189e6aa542a283ea02ba3c9d5: Status 404 returned error can't find the container with id 860d70284c8731717f847d760ad4cb1eafd2b58189e6aa542a283ea02ba3c9d5
Jul 29 14:35:00.843993 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.843913    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 14:35:00.944026 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.943980    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad392a5c-af96-47d5-acdd-9905fc1cf0d5-metrics-certs\") pod \"network-metrics-daemon-vslsw\" (UID: \"ad392a5c-af96-47d5-acdd-9905fc1cf0d5\") " pod="openshift-multus/network-metrics-daemon-vslsw"
Jul 29 14:35:00.944208 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:00.944036    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95701def-c5e7-4309-a07b-6a7c05316dd6-original-pull-secret\") pod \"global-pull-secret-syncer-vg8f5\" (UID: \"95701def-c5e7-4309-a07b-6a7c05316dd6\") " pod="kube-system/global-pull-secret-syncer-vg8f5"
Jul 29 14:35:00.944208 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:00.944144    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 14:35:00.944208 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:00.944175    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 14:35:00.944346 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:00.944225    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95701def-c5e7-4309-a07b-6a7c05316dd6-original-pull-secret podName:95701def-c5e7-4309-a07b-6a7c05316dd6 nodeName:}" failed. No retries permitted until 2026-07-29 14:35:01.944207849 +0000 UTC m=+3.992665767 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/95701def-c5e7-4309-a07b-6a7c05316dd6-original-pull-secret") pod "global-pull-secret-syncer-vg8f5" (UID: "95701def-c5e7-4309-a07b-6a7c05316dd6") : object "kube-system"/"original-pull-secret" not registered
Jul 29 14:35:00.944346 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:00.944242    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad392a5c-af96-47d5-acdd-9905fc1cf0d5-metrics-certs podName:ad392a5c-af96-47d5-acdd-9905fc1cf0d5 nodeName:}" failed. No retries permitted until 2026-07-29 14:35:01.944235215 +0000 UTC m=+3.992693133 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ad392a5c-af96-47d5-acdd-9905fc1cf0d5-metrics-certs") pod "network-metrics-daemon-vslsw" (UID: "ad392a5c-af96-47d5-acdd-9905fc1cf0d5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 14:35:01.044661 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:01.044613    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wltn7\" (UniqueName: \"kubernetes.io/projected/e0828b12-3cef-4e42-8465-bf3dee289acb-kube-api-access-wltn7\") pod \"network-check-target-zvbtr\" (UID: \"e0828b12-3cef-4e42-8465-bf3dee289acb\") " pod="openshift-network-diagnostics/network-check-target-zvbtr"
Jul 29 14:35:01.044814 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:01.044793    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 14:35:01.044864 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:01.044823    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 14:35:01.044864 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:01.044838    2563 projected.go:196] Error preparing data for projected volume kube-api-access-wltn7 for pod openshift-network-diagnostics/network-check-target-zvbtr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 14:35:01.044953 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:01.044900    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0828b12-3cef-4e42-8465-bf3dee289acb-kube-api-access-wltn7 podName:e0828b12-3cef-4e42-8465-bf3dee289acb nodeName:}" failed. No retries permitted until 2026-07-29 14:35:02.044884851 +0000 UTC m=+4.093342769 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wltn7" (UniqueName: "kubernetes.io/projected/e0828b12-3cef-4e42-8465-bf3dee289acb-kube-api-access-wltn7") pod "network-check-target-zvbtr" (UID: "e0828b12-3cef-4e42-8465-bf3dee289acb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 14:35:01.374709 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:01.374602    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 14:30:00 +0000 UTC" deadline="2028-04-26 02:57:22.559273356 +0000 UTC"
Jul 29 14:35:01.374709 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:01.374659    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15276h22m21.184618988s"
Jul 29 14:35:01.444588 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:01.444423    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rzldt" event={"ID":"4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa","Type":"ContainerStarted","Data":"5be6bce154ad5c4aad3d4b1713c09716e7ac5ab206df6d5c3cda38e07ab75788"}
Jul 29 14:35:01.449108 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:01.449044    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k2jpl" event={"ID":"9a100b51-c157-4a63-a6c5-9d92a3fee8a7","Type":"ContainerStarted","Data":"bd4ab80e0cdd10597d08f783650c4ba3c5bac6f713ec763fa1bf775530e19c1b"}
Jul 29 14:35:01.454206 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:01.453611    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l9xjd" event={"ID":"cc5fa7a4-7058-45a4-a2f6-e3a260855a25","Type":"ContainerStarted","Data":"cee43bbbe60553d64d45f9f0031c762307f1fc26c1d335c53dbfc791dabad67c"}
Jul 29 14:35:01.461997 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:01.461900    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5" event={"ID":"c3ea9712-89d4-4d38-9b76-74f7c387f537","Type":"ContainerStarted","Data":"08baa13cc74b5049b95683fae67094a4adf611e88131b6990e25f199848d0ee3"}
Jul 29 14:35:01.465362 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:01.465332    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jsfjb" event={"ID":"ede7f878-5cc1-457e-8849-8ed44e5945bb","Type":"ContainerStarted","Data":"e6520a73552332ca56e62aaeec5a4ea1d787736b2cbf01d9cb90a1f68de75766"}
Jul 29 14:35:01.473940 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:01.473853    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nwj5f" event={"ID":"e666f634-f60c-47fb-952c-7e18a781609b","Type":"ContainerStarted","Data":"aaa62b71ebab08bcfcba4423c36c9d79044ed81c56748900cd1d9f43adfa1911"}
Jul 29 14:35:01.477996 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:01.477732    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d44hd" event={"ID":"75a67018-8010-4684-ac1c-4a12c7f30bea","Type":"ContainerStarted","Data":"60210ed630035a0e5406bb31bb40c5f75c4022d913091c181e3105e60eb5fbf6"}
Jul 29 14:35:01.480516 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:01.479692    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bh4w5" event={"ID":"8dfac3ca-72cd-4bf4-8309-3774c7abe57a","Type":"ContainerStarted","Data":"860d70284c8731717f847d760ad4cb1eafd2b58189e6aa542a283ea02ba3c9d5"}
Jul 29 14:35:01.487242 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:01.484223    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n7qwt" event={"ID":"8a010eab-ffa4-4f5a-a535-ecae3e55cf16","Type":"ContainerStarted","Data":"cf6bc0ef1427d8af39be02b2a0440785f8aaf259fbe0cec72d8e94c91b3803ea"}
Jul 29 14:35:01.952528 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:01.952497    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad392a5c-af96-47d5-acdd-9905fc1cf0d5-metrics-certs\") pod \"network-metrics-daemon-vslsw\" (UID: \"ad392a5c-af96-47d5-acdd-9905fc1cf0d5\") " pod="openshift-multus/network-metrics-daemon-vslsw"
Jul 29 14:35:01.952690 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:01.952561    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95701def-c5e7-4309-a07b-6a7c05316dd6-original-pull-secret\") pod \"global-pull-secret-syncer-vg8f5\" (UID: \"95701def-c5e7-4309-a07b-6a7c05316dd6\") " pod="kube-system/global-pull-secret-syncer-vg8f5"
Jul 29 14:35:01.952755 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:01.952687    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 14:35:01.952755 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:01.952744    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95701def-c5e7-4309-a07b-6a7c05316dd6-original-pull-secret podName:95701def-c5e7-4309-a07b-6a7c05316dd6 nodeName:}" failed. No retries permitted until 2026-07-29 14:35:03.952725167 +0000 UTC m=+6.001183092 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/95701def-c5e7-4309-a07b-6a7c05316dd6-original-pull-secret") pod "global-pull-secret-syncer-vg8f5" (UID: "95701def-c5e7-4309-a07b-6a7c05316dd6") : object "kube-system"/"original-pull-secret" not registered
Jul 29 14:35:01.952878 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:01.952810    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 14:35:01.952878 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:01.952845    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad392a5c-af96-47d5-acdd-9905fc1cf0d5-metrics-certs podName:ad392a5c-af96-47d5-acdd-9905fc1cf0d5 nodeName:}" failed. No retries permitted until 2026-07-29 14:35:03.952833956 +0000 UTC m=+6.001291879 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ad392a5c-af96-47d5-acdd-9905fc1cf0d5-metrics-certs") pod "network-metrics-daemon-vslsw" (UID: "ad392a5c-af96-47d5-acdd-9905fc1cf0d5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 14:35:02.053601 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:02.053562    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wltn7\" (UniqueName: \"kubernetes.io/projected/e0828b12-3cef-4e42-8465-bf3dee289acb-kube-api-access-wltn7\") pod \"network-check-target-zvbtr\" (UID: \"e0828b12-3cef-4e42-8465-bf3dee289acb\") " pod="openshift-network-diagnostics/network-check-target-zvbtr"
Jul 29 14:35:02.053789 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:02.053759    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 14:35:02.053789 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:02.053787    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 14:35:02.053899 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:02.053802    2563 projected.go:196] Error preparing data for projected volume kube-api-access-wltn7 for pod openshift-network-diagnostics/network-check-target-zvbtr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 14:35:02.053899 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:02.053869    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0828b12-3cef-4e42-8465-bf3dee289acb-kube-api-access-wltn7 podName:e0828b12-3cef-4e42-8465-bf3dee289acb nodeName:}" failed. No retries permitted until 2026-07-29 14:35:04.053849756 +0000 UTC m=+6.102307680 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wltn7" (UniqueName: "kubernetes.io/projected/e0828b12-3cef-4e42-8465-bf3dee289acb-kube-api-access-wltn7") pod "network-check-target-zvbtr" (UID: "e0828b12-3cef-4e42-8465-bf3dee289acb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 14:35:02.376157 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:02.376073    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 14:30:00 +0000 UTC" deadline="2028-01-06 04:10:13.174854099 +0000 UTC"
Jul 29 14:35:02.376157 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:02.376117    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12613h35m10.7987435s"
Jul 29 14:35:02.427778 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:02.427287    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vg8f5"
Jul 29 14:35:02.427778 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:02.427326    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvbtr"
Jul 29 14:35:02.427778 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:02.427331    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vslsw"
Jul 29 14:35:02.427778 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:02.427421    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vg8f5" podUID="95701def-c5e7-4309-a07b-6a7c05316dd6"
Jul 29 14:35:02.427778 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:02.427519    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvbtr" podUID="e0828b12-3cef-4e42-8465-bf3dee289acb"
Jul 29 14:35:02.427778 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:02.427616    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vslsw" podUID="ad392a5c-af96-47d5-acdd-9905fc1cf0d5"
Jul 29 14:35:02.523664 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:02.523622    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-196.ec2.internal" event={"ID":"69153ffb63befe8c682d3ac1dcc08c24","Type":"ContainerStarted","Data":"e8e856e01f0753b9e6e39e861c3a8452158b1b90503a9735a0fd5bd118e648c6"}
Jul 29 14:35:03.976447 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:03.976168    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad392a5c-af96-47d5-acdd-9905fc1cf0d5-metrics-certs\") pod \"network-metrics-daemon-vslsw\" (UID: \"ad392a5c-af96-47d5-acdd-9905fc1cf0d5\") " pod="openshift-multus/network-metrics-daemon-vslsw"
Jul 29 14:35:03.976959 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:03.976504    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95701def-c5e7-4309-a07b-6a7c05316dd6-original-pull-secret\") pod \"global-pull-secret-syncer-vg8f5\" (UID: \"95701def-c5e7-4309-a07b-6a7c05316dd6\") " pod="kube-system/global-pull-secret-syncer-vg8f5"
Jul 29 14:35:03.976959 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:03.976674    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 14:35:03.976959 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:03.976749    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95701def-c5e7-4309-a07b-6a7c05316dd6-original-pull-secret podName:95701def-c5e7-4309-a07b-6a7c05316dd6 nodeName:}" failed. No retries permitted until 2026-07-29 14:35:07.976726545 +0000 UTC m=+10.025184489 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/95701def-c5e7-4309-a07b-6a7c05316dd6-original-pull-secret") pod "global-pull-secret-syncer-vg8f5" (UID: "95701def-c5e7-4309-a07b-6a7c05316dd6") : object "kube-system"/"original-pull-secret" not registered
Jul 29 14:35:03.977485 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:03.977240    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 14:35:03.977485 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:03.977302    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad392a5c-af96-47d5-acdd-9905fc1cf0d5-metrics-certs podName:ad392a5c-af96-47d5-acdd-9905fc1cf0d5 nodeName:}" failed. No retries permitted until 2026-07-29 14:35:07.97728421 +0000 UTC m=+10.025742136 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ad392a5c-af96-47d5-acdd-9905fc1cf0d5-metrics-certs") pod "network-metrics-daemon-vslsw" (UID: "ad392a5c-af96-47d5-acdd-9905fc1cf0d5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 14:35:04.078172 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:04.077478    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wltn7\" (UniqueName: \"kubernetes.io/projected/e0828b12-3cef-4e42-8465-bf3dee289acb-kube-api-access-wltn7\") pod \"network-check-target-zvbtr\" (UID: \"e0828b12-3cef-4e42-8465-bf3dee289acb\") " pod="openshift-network-diagnostics/network-check-target-zvbtr"
Jul 29 14:35:04.078172 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:04.077679    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 14:35:04.078172 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:04.077702    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 14:35:04.078172 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:04.077718    2563 projected.go:196] Error preparing data for projected volume kube-api-access-wltn7 for pod openshift-network-diagnostics/network-check-target-zvbtr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 14:35:04.078172 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:04.077776    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0828b12-3cef-4e42-8465-bf3dee289acb-kube-api-access-wltn7 podName:e0828b12-3cef-4e42-8465-bf3dee289acb nodeName:}" failed. No retries permitted until 2026-07-29 14:35:08.077757227 +0000 UTC m=+10.126215156 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wltn7" (UniqueName: "kubernetes.io/projected/e0828b12-3cef-4e42-8465-bf3dee289acb-kube-api-access-wltn7") pod "network-check-target-zvbtr" (UID: "e0828b12-3cef-4e42-8465-bf3dee289acb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 14:35:04.428443 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:04.428365    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vslsw"
Jul 29 14:35:04.428611 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:04.428522    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vslsw" podUID="ad392a5c-af96-47d5-acdd-9905fc1cf0d5"
Jul 29 14:35:04.428795 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:04.428722    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvbtr"
Jul 29 14:35:04.428909 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:04.428840    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvbtr" podUID="e0828b12-3cef-4e42-8465-bf3dee289acb"
Jul 29 14:35:04.428982 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:04.428913    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vg8f5"
Jul 29 14:35:04.429053 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:04.428992    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vg8f5" podUID="95701def-c5e7-4309-a07b-6a7c05316dd6"
Jul 29 14:35:06.426481 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:06.426453    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vg8f5"
Jul 29 14:35:06.426962 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:06.426580    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vg8f5" podUID="95701def-c5e7-4309-a07b-6a7c05316dd6"
Jul 29 14:35:06.427051 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:06.426962    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvbtr"
Jul 29 14:35:06.427093 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:06.427053    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvbtr" podUID="e0828b12-3cef-4e42-8465-bf3dee289acb"
Jul 29 14:35:06.427367 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:06.427165    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vslsw"
Jul 29 14:35:06.427367 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:06.427328    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vslsw" podUID="ad392a5c-af96-47d5-acdd-9905fc1cf0d5"
Jul 29 14:35:06.539333 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:06.539296    2563 generic.go:356] "Generic (PLEG): container finished" podID="69153ffb63befe8c682d3ac1dcc08c24" containerID="e8e856e01f0753b9e6e39e861c3a8452158b1b90503a9735a0fd5bd118e648c6" exitCode=0
Jul 29 14:35:06.539558 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:06.539354    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-196.ec2.internal" event={"ID":"69153ffb63befe8c682d3ac1dcc08c24","Type":"ContainerDied","Data":"e8e856e01f0753b9e6e39e861c3a8452158b1b90503a9735a0fd5bd118e648c6"}
Jul 29 14:35:08.014360 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:08.013716    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95701def-c5e7-4309-a07b-6a7c05316dd6-original-pull-secret\") pod \"global-pull-secret-syncer-vg8f5\" (UID: \"95701def-c5e7-4309-a07b-6a7c05316dd6\") " pod="kube-system/global-pull-secret-syncer-vg8f5"
Jul 29 14:35:08.014360 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:08.013807    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad392a5c-af96-47d5-acdd-9905fc1cf0d5-metrics-certs\") pod \"network-metrics-daemon-vslsw\" (UID: \"ad392a5c-af96-47d5-acdd-9905fc1cf0d5\") " pod="openshift-multus/network-metrics-daemon-vslsw"
Jul 29 14:35:08.014360 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:08.013906    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 14:35:08.014360 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:08.013972    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad392a5c-af96-47d5-acdd-9905fc1cf0d5-metrics-certs podName:ad392a5c-af96-47d5-acdd-9905fc1cf0d5 nodeName:}" failed. No retries permitted until 2026-07-29 14:35:16.013951633 +0000 UTC m=+18.062409556 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ad392a5c-af96-47d5-acdd-9905fc1cf0d5-metrics-certs") pod "network-metrics-daemon-vslsw" (UID: "ad392a5c-af96-47d5-acdd-9905fc1cf0d5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 14:35:08.014360 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:08.013907    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 14:35:08.014360 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:08.014098    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95701def-c5e7-4309-a07b-6a7c05316dd6-original-pull-secret podName:95701def-c5e7-4309-a07b-6a7c05316dd6 nodeName:}" failed. No retries permitted until 2026-07-29 14:35:16.014076474 +0000 UTC m=+18.062534402 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/95701def-c5e7-4309-a07b-6a7c05316dd6-original-pull-secret") pod "global-pull-secret-syncer-vg8f5" (UID: "95701def-c5e7-4309-a07b-6a7c05316dd6") : object "kube-system"/"original-pull-secret" not registered
Jul 29 14:35:08.114602 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:08.114495    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wltn7\" (UniqueName: \"kubernetes.io/projected/e0828b12-3cef-4e42-8465-bf3dee289acb-kube-api-access-wltn7\") pod \"network-check-target-zvbtr\" (UID: \"e0828b12-3cef-4e42-8465-bf3dee289acb\") " pod="openshift-network-diagnostics/network-check-target-zvbtr"
Jul 29 14:35:08.114752 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:08.114660    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 14:35:08.114752 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:08.114681    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 14:35:08.114752 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:08.114693    2563 projected.go:196] Error preparing data for projected volume kube-api-access-wltn7 for pod openshift-network-diagnostics/network-check-target-zvbtr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 14:35:08.114752 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:08.114741    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0828b12-3cef-4e42-8465-bf3dee289acb-kube-api-access-wltn7 podName:e0828b12-3cef-4e42-8465-bf3dee289acb nodeName:}" failed. No retries permitted until 2026-07-29 14:35:16.114727816 +0000 UTC m=+18.163185733 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wltn7" (UniqueName: "kubernetes.io/projected/e0828b12-3cef-4e42-8465-bf3dee289acb-kube-api-access-wltn7") pod "network-check-target-zvbtr" (UID: "e0828b12-3cef-4e42-8465-bf3dee289acb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 14:35:08.427668 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:08.427577    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vg8f5"
Jul 29 14:35:08.427867 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:08.427699    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vg8f5" podUID="95701def-c5e7-4309-a07b-6a7c05316dd6"
Jul 29 14:35:08.428428 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:08.428124    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvbtr"
Jul 29 14:35:08.428428 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:08.428237    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvbtr" podUID="e0828b12-3cef-4e42-8465-bf3dee289acb"
Jul 29 14:35:08.428428 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:08.428306    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vslsw"
Jul 29 14:35:08.428428 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:08.428389    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vslsw" podUID="ad392a5c-af96-47d5-acdd-9905fc1cf0d5"
Jul 29 14:35:10.426737 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:10.426224    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vg8f5"
Jul 29 14:35:10.426737 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:10.426239    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvbtr"
Jul 29 14:35:10.426737 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:10.426361    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vg8f5" podUID="95701def-c5e7-4309-a07b-6a7c05316dd6"
Jul 29 14:35:10.426737 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:10.426409    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vslsw"
Jul 29 14:35:10.426737 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:10.426495    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvbtr" podUID="e0828b12-3cef-4e42-8465-bf3dee289acb"
Jul 29 14:35:10.426737 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:10.426620    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vslsw" podUID="ad392a5c-af96-47d5-acdd-9905fc1cf0d5"
Jul 29 14:35:12.426233 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:12.426192    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vg8f5"
Jul 29 14:35:12.426645 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:12.426245    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvbtr"
Jul 29 14:35:12.426645 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:12.426192    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vslsw"
Jul 29 14:35:12.426645 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:12.426334    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vg8f5" podUID="95701def-c5e7-4309-a07b-6a7c05316dd6"
Jul 29 14:35:12.426645 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:12.426431    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvbtr" podUID="e0828b12-3cef-4e42-8465-bf3dee289acb"
Jul 29 14:35:12.426645 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:12.426510    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vslsw" podUID="ad392a5c-af96-47d5-acdd-9905fc1cf0d5"
Jul 29 14:35:14.426150 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:14.426109    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvbtr"
Jul 29 14:35:14.426632 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:14.426109    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vg8f5"
Jul 29 14:35:14.426632 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:14.426243    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvbtr" podUID="e0828b12-3cef-4e42-8465-bf3dee289acb"
Jul 29 14:35:14.426632 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:14.426113    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vslsw"
Jul 29 14:35:14.426632 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:14.426328    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vg8f5" podUID="95701def-c5e7-4309-a07b-6a7c05316dd6"
Jul 29 14:35:14.426632 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:14.426420    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vslsw" podUID="ad392a5c-af96-47d5-acdd-9905fc1cf0d5"
Jul 29 14:35:16.070560 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:16.070526    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad392a5c-af96-47d5-acdd-9905fc1cf0d5-metrics-certs\") pod \"network-metrics-daemon-vslsw\" (UID: \"ad392a5c-af96-47d5-acdd-9905fc1cf0d5\") " pod="openshift-multus/network-metrics-daemon-vslsw"
Jul 29 14:35:16.070973 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:16.070571    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95701def-c5e7-4309-a07b-6a7c05316dd6-original-pull-secret\") pod \"global-pull-secret-syncer-vg8f5\" (UID: \"95701def-c5e7-4309-a07b-6a7c05316dd6\") " pod="kube-system/global-pull-secret-syncer-vg8f5"
Jul 29 14:35:16.070973 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:16.070721    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 14:35:16.070973 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:16.070792    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad392a5c-af96-47d5-acdd-9905fc1cf0d5-metrics-certs podName:ad392a5c-af96-47d5-acdd-9905fc1cf0d5 nodeName:}" failed. No retries permitted until 2026-07-29 14:35:32.070775947 +0000 UTC m=+34.119233864 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ad392a5c-af96-47d5-acdd-9905fc1cf0d5-metrics-certs") pod "network-metrics-daemon-vslsw" (UID: "ad392a5c-af96-47d5-acdd-9905fc1cf0d5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 14:35:16.070973 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:16.070727    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 14:35:16.070973 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:16.070857    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95701def-c5e7-4309-a07b-6a7c05316dd6-original-pull-secret podName:95701def-c5e7-4309-a07b-6a7c05316dd6 nodeName:}" failed. No retries permitted until 2026-07-29 14:35:32.07084338 +0000 UTC m=+34.119301304 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/95701def-c5e7-4309-a07b-6a7c05316dd6-original-pull-secret") pod "global-pull-secret-syncer-vg8f5" (UID: "95701def-c5e7-4309-a07b-6a7c05316dd6") : object "kube-system"/"original-pull-secret" not registered
Jul 29 14:35:16.171047 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:16.171004    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wltn7\" (UniqueName: \"kubernetes.io/projected/e0828b12-3cef-4e42-8465-bf3dee289acb-kube-api-access-wltn7\") pod \"network-check-target-zvbtr\" (UID: \"e0828b12-3cef-4e42-8465-bf3dee289acb\") " pod="openshift-network-diagnostics/network-check-target-zvbtr"
Jul 29 14:35:16.171248 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:16.171174    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 14:35:16.171248 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:16.171196    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 14:35:16.171248 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:16.171205    2563 projected.go:196] Error preparing data for projected volume kube-api-access-wltn7 for pod openshift-network-diagnostics/network-check-target-zvbtr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 14:35:16.171357 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:16.171265    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0828b12-3cef-4e42-8465-bf3dee289acb-kube-api-access-wltn7 podName:e0828b12-3cef-4e42-8465-bf3dee289acb nodeName:}" failed. No retries permitted until 2026-07-29 14:35:32.171249763 +0000 UTC m=+34.219707703 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wltn7" (UniqueName: "kubernetes.io/projected/e0828b12-3cef-4e42-8465-bf3dee289acb-kube-api-access-wltn7") pod "network-check-target-zvbtr" (UID: "e0828b12-3cef-4e42-8465-bf3dee289acb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 14:35:16.426560 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:16.426468    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvbtr"
Jul 29 14:35:16.426560 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:16.426509    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vslsw"
Jul 29 14:35:16.426560 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:16.426509    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vg8f5"
Jul 29 14:35:16.426832 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:16.426618    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvbtr" podUID="e0828b12-3cef-4e42-8465-bf3dee289acb"
Jul 29 14:35:16.426832 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:16.426694    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vslsw" podUID="ad392a5c-af96-47d5-acdd-9905fc1cf0d5"
Jul 29 14:35:16.426832 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:16.426784    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vg8f5" podUID="95701def-c5e7-4309-a07b-6a7c05316dd6"
Jul 29 14:35:18.427214 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:18.427178    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vslsw"
Jul 29 14:35:18.427214 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:18.427210    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vg8f5"
Jul 29 14:35:18.427744 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:18.427308    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vslsw" podUID="ad392a5c-af96-47d5-acdd-9905fc1cf0d5"
Jul 29 14:35:18.427744 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:18.427369    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vg8f5" podUID="95701def-c5e7-4309-a07b-6a7c05316dd6"
Jul 29 14:35:18.427744 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:18.427431    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvbtr"
Jul 29 14:35:18.427744 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:18.427532    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvbtr" podUID="e0828b12-3cef-4e42-8465-bf3dee289acb"
Jul 29 14:35:20.426363 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:20.426288    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vg8f5"
Jul 29 14:35:20.426708 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:20.426288    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvbtr"
Jul 29 14:35:20.426708 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:20.426409    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vg8f5" podUID="95701def-c5e7-4309-a07b-6a7c05316dd6"
Jul 29 14:35:20.426708 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:20.426288    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vslsw"
Jul 29 14:35:20.426708 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:20.426494    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvbtr" podUID="e0828b12-3cef-4e42-8465-bf3dee289acb"
Jul 29 14:35:20.426708 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:20.426582    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vslsw" podUID="ad392a5c-af96-47d5-acdd-9905fc1cf0d5"
Jul 29 14:35:21.575175 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:21.574841    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-196.ec2.internal" event={"ID":"69153ffb63befe8c682d3ac1dcc08c24","Type":"ContainerStarted","Data":"e1879cee9679af6b3501d2b3fec7833e64a0ca12747b4642bb89863f904d656c"}
Jul 29 14:35:21.576314 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:21.576291    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rzldt" event={"ID":"4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa","Type":"ContainerStarted","Data":"694353253fc1f4117552ab18455603d31618c0a7eb5516c6fe113ce748debaf1"}
Jul 29 14:35:21.577522 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:21.577502    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k2jpl" event={"ID":"9a100b51-c157-4a63-a6c5-9d92a3fee8a7","Type":"ContainerStarted","Data":"487ae380a298829aaca64b3da2a785d888fb0d7817b7739a2320ad5fea260773"}
Jul 29 14:35:21.580145 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:21.580114    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5" event={"ID":"c3ea9712-89d4-4d38-9b76-74f7c387f537","Type":"ContainerStarted","Data":"8217e6587950171d7bb9999e58ca5b8a870a049e2e2bcdbcaddd0607fe676ce3"}
Jul 29 14:35:21.580241 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:21.580196    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5" event={"ID":"c3ea9712-89d4-4d38-9b76-74f7c387f537","Type":"ContainerStarted","Data":"8dc20e28af26abce487e2ce9ce2eb73579b953e7de22cb83790082a23c51bfb3"}
Jul 29 14:35:21.580286 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:21.580247    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5" event={"ID":"c3ea9712-89d4-4d38-9b76-74f7c387f537","Type":"ContainerStarted","Data":"3eb97933ad58912a5dd45ed2ad6f564f96e4fbf96fbd20dbf4ff216c4ec3e8dc"}
Jul 29 14:35:21.580319 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:21.580295    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5" event={"ID":"c3ea9712-89d4-4d38-9b76-74f7c387f537","Type":"ContainerStarted","Data":"aaf5f33e421cd566c67e1d4c34edd32be7234f58d57bd069ff817261a532cb2e"}
Jul 29 14:35:21.580351 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:21.580341    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5" event={"ID":"c3ea9712-89d4-4d38-9b76-74f7c387f537","Type":"ContainerStarted","Data":"a0b190acc549ab88a785b1efa2a8926a92515fef64f038c43edfb5b80b71bd8f"}
Jul 29 14:35:21.580387 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:21.580380    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5" event={"ID":"c3ea9712-89d4-4d38-9b76-74f7c387f537","Type":"ContainerStarted","Data":"091e264e4e22603f3f60919b7211aec19544ddcbb728f2b563de1f569932af37"}
Jul 29 14:35:21.581493 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:21.581475    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jsfjb" event={"ID":"ede7f878-5cc1-457e-8849-8ed44e5945bb","Type":"ContainerStarted","Data":"504724ade67c8a099b3d0910872de9f29efaac6f64e88b4994448f9c02ec6e44"}
Jul 29 14:35:21.582847 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:21.582829    2563 generic.go:356] "Generic (PLEG): container finished" podID="e666f634-f60c-47fb-952c-7e18a781609b" containerID="d069bd01fccdf19614a4dbe72f4fcd6d84f320e97e3775a7a59382261f64ca61" exitCode=0
Jul 29 14:35:21.582925 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:21.582884    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nwj5f" event={"ID":"e666f634-f60c-47fb-952c-7e18a781609b","Type":"ContainerDied","Data":"d069bd01fccdf19614a4dbe72f4fcd6d84f320e97e3775a7a59382261f64ca61"}
Jul 29 14:35:21.584266 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:21.584243    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d44hd" event={"ID":"75a67018-8010-4684-ac1c-4a12c7f30bea","Type":"ContainerStarted","Data":"adb9671de32375c323cd63a6a6ff8dfd1a8ded4b043357e77337dd883895a3c4"}
Jul 29 14:35:21.585564 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:21.585544    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-196.ec2.internal" event={"ID":"c88b7e35b1a8b1d19afb95df94d0ce5f","Type":"ContainerStarted","Data":"e5c970fec4ca849ac923debcc56ab3b75cc62933c0a317745a3df89a5834e572"}
Jul 29 14:35:21.586833 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:21.586814    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bh4w5" event={"ID":"8dfac3ca-72cd-4bf4-8309-3774c7abe57a","Type":"ContainerStarted","Data":"5cac422a082d67ff4cb541dc27488956711d0f0e192b52435a117f19aa9f5298"}
Jul 29 14:35:21.588151 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:21.588119    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n7qwt" event={"ID":"8a010eab-ffa4-4f5a-a535-ecae3e55cf16","Type":"ContainerStarted","Data":"7d58999f361ac52d5c4b15481b15ce3c067c7df9b7ddbc813e484ab79d408bae"}
Jul 29 14:35:21.589647 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:21.589614    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-196.ec2.internal" podStartSLOduration=21.589602936 podStartE2EDuration="21.589602936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 14:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:35:21.58942334 +0000 UTC m=+23.637881278" watchObservedRunningTime="2026-07-29 14:35:21.589602936 +0000 UTC m=+23.638060869"
Jul 29 14:35:21.605020 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:21.604978    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-196.ec2.internal" podStartSLOduration=21.604965231 podStartE2EDuration="21.604965231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 14:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:35:21.601671767 +0000 UTC m=+23.650129707" watchObservedRunningTime="2026-07-29 14:35:21.604965231 +0000 UTC m=+23.653423169"
Jul 29 14:35:21.614751 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:21.614717    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-d44hd" podStartSLOduration=3.756500399 podStartE2EDuration="23.614705433s" totalImagesPullingTime="19.858205034s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 14:34:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:35:21.614337632 +0000 UTC m=+23.662795573" watchObservedRunningTime="2026-07-29 14:35:21.614705433 +0000 UTC m=+23.663163372"
Jul 29 14:35:21.625974 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:21.625930    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-n7qwt" podStartSLOduration=2.787037721 podStartE2EDuration="22.625918759s" totalImagesPullingTime="19.838881038s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 14:34:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:35:21.625803365 +0000 UTC m=+23.674261303" watchObservedRunningTime="2026-07-29 14:35:21.625918759 +0000 UTC m=+23.674376695"
Jul 29 14:35:21.657769 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:21.657725    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-k2jpl" podStartSLOduration=2.938137618 podStartE2EDuration="22.657709155s" totalImagesPullingTime="19.719571537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:34:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:35:21.639164322 +0000 UTC m=+23.687622257" watchObservedRunningTime="2026-07-29 14:35:21.657709155 +0000 UTC m=+23.706167093"
Jul 29 14:35:21.674182 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:21.674121    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jsfjb" podStartSLOduration=3.802970635 podStartE2EDuration="23.674104263s" totalImagesPullingTime="19.871133628s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 14:34:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:35:21.673524785 +0000 UTC m=+23.721982725" watchObservedRunningTime="2026-07-29 14:35:21.674104263 +0000 UTC m=+23.722562201"
Jul 29 14:35:21.688920 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:21.688882    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bh4w5" podStartSLOduration=2.87184944 podStartE2EDuration="22.688868691s" totalImagesPullingTime="19.817019251s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 14:34:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:35:21.688724698 +0000 UTC m=+23.737182638" watchObservedRunningTime="2026-07-29 14:35:21.688868691 +0000 UTC m=+23.737326916"
Jul 29 14:35:22.426544 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:22.426509    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vg8f5"
Jul 29 14:35:22.426802 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:22.426509    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvbtr"
Jul 29 14:35:22.426802 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:22.426603    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vg8f5" podUID="95701def-c5e7-4309-a07b-6a7c05316dd6"
Jul 29 14:35:22.426802 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:22.426619    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vslsw"
Jul 29 14:35:22.426802 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:22.426709    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvbtr" podUID="e0828b12-3cef-4e42-8465-bf3dee289acb"
Jul 29 14:35:22.426946 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:22.426813    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vslsw" podUID="ad392a5c-af96-47d5-acdd-9905fc1cf0d5"
Jul 29 14:35:22.574326 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:22.574295    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 29 14:35:22.591563 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:22.591537    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rzldt" event={"ID":"4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa","Type":"ContainerStarted","Data":"e814c7dc32437d55d826d131651102b02e900a6ad6bc7849d41d78f4c11eeee1"}
Jul 29 14:35:22.592851 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:22.592833    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l9xjd" event={"ID":"cc5fa7a4-7058-45a4-a2f6-e3a260855a25","Type":"ContainerStarted","Data":"0d22792e6b7cf23733d60b16e3dacdc2ec068fa0c8501dc3abd347df2310d46e"}
Jul 29 14:35:22.605440 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:22.605405    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-l9xjd" podStartSLOduration=3.772579223 podStartE2EDuration="23.605393459s" totalImagesPullingTime="19.832814236s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 14:34:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:35:22.605315967 +0000 UTC m=+24.653773906" watchObservedRunningTime="2026-07-29 14:35:22.605393459 +0000 UTC m=+24.653851398"
Jul 29 14:35:23.387447 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:23.387376    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T14:35:22.574317477Z","UUID":"d85ced55-898d-4b46-aabe-1aca22cd38ab","Handler":null,"Name":"","Endpoint":""}
Jul 29 14:35:23.388713 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:23.388696    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 29 14:35:23.388817 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:23.388731    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 29 14:35:23.599036 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:23.598956    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5" event={"ID":"c3ea9712-89d4-4d38-9b76-74f7c387f537","Type":"ContainerStarted","Data":"6eb762fe69ad20f50b7a80e65ef1370272dff09c60c46c7ea9bfaac017ab0527"}
Jul 29 14:35:24.426049 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:24.426018    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vslsw"
Jul 29 14:35:24.426242 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:24.426017    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvbtr"
Jul 29 14:35:24.426242 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:24.426168    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vslsw" podUID="ad392a5c-af96-47d5-acdd-9905fc1cf0d5"
Jul 29 14:35:24.426374 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:24.426235    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvbtr" podUID="e0828b12-3cef-4e42-8465-bf3dee289acb"
Jul 29 14:35:24.426374 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:24.426018    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vg8f5"
Jul 29 14:35:24.426374 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:24.426333    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vg8f5" podUID="95701def-c5e7-4309-a07b-6a7c05316dd6"
Jul 29 14:35:25.431151 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:25.430898    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-k2jpl"
Jul 29 14:35:25.431606 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:25.431576    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-k2jpl"
Jul 29 14:35:25.605724 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:25.605680    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rzldt" event={"ID":"4f2d7ae6-f34b-4d7e-be9a-587701d9c0aa","Type":"ContainerStarted","Data":"2c22d8267410cdd1b098e9da807949796b62a3dbf1b7d93ce1a011cfceade80b"}
Jul 29 14:35:25.605906 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:25.605838    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-k2jpl"
Jul 29 14:35:25.606433 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:25.606405    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-k2jpl"
Jul 29 14:35:25.624052 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:25.624010    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rzldt" podStartSLOduration=2.8434612059999997 podStartE2EDuration="26.62399674s" totalImagesPullingTime="23.780535534s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 14:34:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:35:25.623519078 +0000 UTC m=+27.671977019" watchObservedRunningTime="2026-07-29 14:35:25.62399674 +0000 UTC m=+27.672454679"
Jul 29 14:35:26.426547 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:26.426511    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvbtr"
Jul 29 14:35:26.426547 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:26.426544    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vg8f5"
Jul 29 14:35:26.426778 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:26.426518    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vslsw"
Jul 29 14:35:26.426778 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:26.426628    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvbtr" podUID="e0828b12-3cef-4e42-8465-bf3dee289acb"
Jul 29 14:35:26.426778 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:26.426721    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vslsw" podUID="ad392a5c-af96-47d5-acdd-9905fc1cf0d5"
Jul 29 14:35:26.426778 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:26.426784    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vg8f5" podUID="95701def-c5e7-4309-a07b-6a7c05316dd6"
Jul 29 14:35:27.611983 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:27.611752    2563 generic.go:356] "Generic (PLEG): container finished" podID="e666f634-f60c-47fb-952c-7e18a781609b" containerID="46847c401f9894102873405b1f3c59693ad9b2a713bf45a27e0e4454475ecb02" exitCode=0
Jul 29 14:35:27.611983 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:27.611814    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nwj5f" event={"ID":"e666f634-f60c-47fb-952c-7e18a781609b","Type":"ContainerDied","Data":"46847c401f9894102873405b1f3c59693ad9b2a713bf45a27e0e4454475ecb02"}
Jul 29 14:35:27.615751 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:27.615723    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5" event={"ID":"c3ea9712-89d4-4d38-9b76-74f7c387f537","Type":"ContainerStarted","Data":"e239c054e0e025af26acbd48d4059c81820d11d93ccd4fec6e79446867d0ba16"}
Jul 29 14:35:27.651964 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:27.651873    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5" podStartSLOduration=8.668991706 podStartE2EDuration="28.651854622s" totalImagesPullingTime="19.982862916s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 14:34:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:35:27.651709886 +0000 UTC m=+29.700167824" watchObservedRunningTime="2026-07-29 14:35:27.651854622 +0000 UTC m=+29.700312562"
Jul 29 14:35:28.430167 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:28.430099    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vslsw"
Jul 29 14:35:28.430167 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:28.430116    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vg8f5"
Jul 29 14:35:28.430539 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:28.430503    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vg8f5" podUID="95701def-c5e7-4309-a07b-6a7c05316dd6"
Jul 29 14:35:28.431049 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:28.431004    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vslsw" podUID="ad392a5c-af96-47d5-acdd-9905fc1cf0d5"
Jul 29 14:35:28.431251 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:28.431205    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvbtr"
Jul 29 14:35:28.431356 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:28.431330    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvbtr" podUID="e0828b12-3cef-4e42-8465-bf3dee289acb"
Jul 29 14:35:28.618532 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:28.618498    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:28.618908 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:28.618572    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:28.618908 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:28.618616    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:28.633562 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:28.633540    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:28.633725 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:28.633653    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:35:29.205376 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:29.205352    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vslsw"]
Jul 29 14:35:29.205494 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:29.205457    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vslsw"
Jul 29 14:35:29.205575 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:29.205553    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vslsw" podUID="ad392a5c-af96-47d5-acdd-9905fc1cf0d5"
Jul 29 14:35:29.208317 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:29.208292    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vg8f5"]
Jul 29 14:35:29.208440 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:29.208385    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vg8f5"
Jul 29 14:35:29.208537 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:29.208475    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vg8f5" podUID="95701def-c5e7-4309-a07b-6a7c05316dd6"
Jul 29 14:35:29.209189 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:29.209165    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zvbtr"]
Jul 29 14:35:29.209293 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:29.209246    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvbtr"
Jul 29 14:35:29.209397 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:29.209369    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvbtr" podUID="e0828b12-3cef-4e42-8465-bf3dee289acb"
Jul 29 14:35:29.622612 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:29.622581    2563 generic.go:356] "Generic (PLEG): container finished" podID="e666f634-f60c-47fb-952c-7e18a781609b" containerID="142084802a0b16ea7db2567d899531310f28a622961398ab95ba96b63e3e6e94" exitCode=0
Jul 29 14:35:29.622954 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:29.622659    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nwj5f" event={"ID":"e666f634-f60c-47fb-952c-7e18a781609b","Type":"ContainerDied","Data":"142084802a0b16ea7db2567d899531310f28a622961398ab95ba96b63e3e6e94"}
Jul 29 14:35:30.426769 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:30.426585    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vg8f5"
Jul 29 14:35:30.426914 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:30.426843    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vg8f5" podUID="95701def-c5e7-4309-a07b-6a7c05316dd6"
Jul 29 14:35:31.426321 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:31.426227    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vslsw"
Jul 29 14:35:31.426871 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:31.426360    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vslsw" podUID="ad392a5c-af96-47d5-acdd-9905fc1cf0d5"
Jul 29 14:35:31.426871 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:31.426408    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvbtr"
Jul 29 14:35:31.426871 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:31.426487    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvbtr" podUID="e0828b12-3cef-4e42-8465-bf3dee289acb"
Jul 29 14:35:31.629382 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:31.629350    2563 generic.go:356] "Generic (PLEG): container finished" podID="e666f634-f60c-47fb-952c-7e18a781609b" containerID="5c11799eb3aaeee5560ef26f4602c0211a835d2241f971b087385c7987aa6503" exitCode=0
Jul 29 14:35:31.629543 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:31.629402    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nwj5f" event={"ID":"e666f634-f60c-47fb-952c-7e18a781609b","Type":"ContainerDied","Data":"5c11799eb3aaeee5560ef26f4602c0211a835d2241f971b087385c7987aa6503"}
Jul 29 14:35:32.096902 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:32.096861    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad392a5c-af96-47d5-acdd-9905fc1cf0d5-metrics-certs\") pod \"network-metrics-daemon-vslsw\" (UID: \"ad392a5c-af96-47d5-acdd-9905fc1cf0d5\") " pod="openshift-multus/network-metrics-daemon-vslsw"
Jul 29 14:35:32.097103 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:32.096922    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95701def-c5e7-4309-a07b-6a7c05316dd6-original-pull-secret\") pod \"global-pull-secret-syncer-vg8f5\" (UID: \"95701def-c5e7-4309-a07b-6a7c05316dd6\") " pod="kube-system/global-pull-secret-syncer-vg8f5"
Jul 29 14:35:32.097103 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:32.097027    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 14:35:32.097103 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:32.097103    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad392a5c-af96-47d5-acdd-9905fc1cf0d5-metrics-certs podName:ad392a5c-af96-47d5-acdd-9905fc1cf0d5 nodeName:}" failed. No retries permitted until 2026-07-29 14:36:04.097083721 +0000 UTC m=+66.145541643 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ad392a5c-af96-47d5-acdd-9905fc1cf0d5-metrics-certs") pod "network-metrics-daemon-vslsw" (UID: "ad392a5c-af96-47d5-acdd-9905fc1cf0d5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 14:35:32.097298 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:32.097031    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 14:35:32.097298 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:32.097240    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95701def-c5e7-4309-a07b-6a7c05316dd6-original-pull-secret podName:95701def-c5e7-4309-a07b-6a7c05316dd6 nodeName:}" failed. No retries permitted until 2026-07-29 14:36:04.097219976 +0000 UTC m=+66.145677893 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/95701def-c5e7-4309-a07b-6a7c05316dd6-original-pull-secret") pod "global-pull-secret-syncer-vg8f5" (UID: "95701def-c5e7-4309-a07b-6a7c05316dd6") : object "kube-system"/"original-pull-secret" not registered
Jul 29 14:35:32.197741 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:32.197657    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wltn7\" (UniqueName: \"kubernetes.io/projected/e0828b12-3cef-4e42-8465-bf3dee289acb-kube-api-access-wltn7\") pod \"network-check-target-zvbtr\" (UID: \"e0828b12-3cef-4e42-8465-bf3dee289acb\") " pod="openshift-network-diagnostics/network-check-target-zvbtr"
Jul 29 14:35:32.197912 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:32.197842    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 14:35:32.197912 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:32.197867    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 14:35:32.197912 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:32.197877    2563 projected.go:196] Error preparing data for projected volume kube-api-access-wltn7 for pod openshift-network-diagnostics/network-check-target-zvbtr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 14:35:32.198027 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:32.197933    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0828b12-3cef-4e42-8465-bf3dee289acb-kube-api-access-wltn7 podName:e0828b12-3cef-4e42-8465-bf3dee289acb nodeName:}" failed. No retries permitted until 2026-07-29 14:36:04.197914324 +0000 UTC m=+66.246372258 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wltn7" (UniqueName: "kubernetes.io/projected/e0828b12-3cef-4e42-8465-bf3dee289acb-kube-api-access-wltn7") pod "network-check-target-zvbtr" (UID: "e0828b12-3cef-4e42-8465-bf3dee289acb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 14:35:32.429522 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:32.429493    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vg8f5"
Jul 29 14:35:32.430013 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:32.429607    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vg8f5" podUID="95701def-c5e7-4309-a07b-6a7c05316dd6"
Jul 29 14:35:33.256302 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.256270    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-196.ec2.internal" event="NodeReady"
Jul 29 14:35:33.256476 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.256416    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 29 14:35:33.294302 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.294243    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-g9dbl"]
Jul 29 14:35:33.318624 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.318548    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-xnm95"]
Jul 29 14:35:33.318780 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.318683    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-g9dbl"
Jul 29 14:35:33.322839 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.322803    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 29 14:35:33.333317 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.333290    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 29 14:35:33.333425 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.333385    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-sgp9t\""
Jul 29 14:35:33.346669 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.346645    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d8957c4d5-4pb5r"]
Jul 29 14:35:33.347287 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.347115    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-xnm95"
Jul 29 14:35:33.351543 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.351522    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 29 14:35:33.351679 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.351658    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 29 14:35:33.351751 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.351689    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 29 14:35:33.352561 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.352542    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-97hvh\""
Jul 29 14:35:33.352683 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.352666    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 29 14:35:33.359539 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.359519    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 29 14:35:33.362482 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.362464    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-g9dbl"]
Jul 29 14:35:33.362572 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.362500    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-xnm95"]
Jul 29 14:35:33.362572 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.362519    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d8957c4d5-4pb5r"]
Jul 29 14:35:33.362572 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.362537    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-nh8pd"]
Jul 29 14:35:33.362743 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.362608    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r"
Jul 29 14:35:33.365653 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.365629    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hfmq9\""
Jul 29 14:35:33.370791 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.370775    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 29 14:35:33.371453 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.371429    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 29 14:35:33.371933 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.371913    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 29 14:35:33.380483 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.380457    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-fkjfz"]
Jul 29 14:35:33.381415 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.381394    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-nh8pd"
Jul 29 14:35:33.382377 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.382288    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 29 14:35:33.384178 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.384123    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 29 14:35:33.384178 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.384170    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 29 14:35:33.384382 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.384247    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-nzvpm\""
Jul 29 14:35:33.384462 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.384443    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 29 14:35:33.384683 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.384668    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 29 14:35:33.389641 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.389619    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 29 14:35:33.405107 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.405075    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-746f69dff9-2bdj7"]
Jul 29 14:35:33.405227 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.405197    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-fkjfz"
Jul 29 14:35:33.405868 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.405805    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65jr9\" (UniqueName: \"kubernetes.io/projected/c1eb675a-8bbd-48ef-a526-506a7fd20ee7-kube-api-access-65jr9\") pod \"volume-data-source-validator-549c6d54cf-g9dbl\" (UID: \"c1eb675a-8bbd-48ef-a526-506a7fd20ee7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-g9dbl"
Jul 29 14:35:33.407405 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.407379    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 29 14:35:33.407610 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.407591    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-d495b\""
Jul 29 14:35:33.407763 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.407729    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 29 14:35:33.407763 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.407741    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 29 14:35:33.407763 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.407759    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 29 14:35:33.424690 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.424666    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-bxjbs"]
Jul 29 14:35:33.424819 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.424782    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-746f69dff9-2bdj7"
Jul 29 14:35:33.430667 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.430647    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 29 14:35:33.431246 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.431226    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 29 14:35:33.431364 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.431293    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8b7g5\""
Jul 29 14:35:33.431364 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.431305    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 29 14:35:33.431512 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.431298    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 29 14:35:33.431512 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.431380    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 29 14:35:33.432536 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.432513    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 29 14:35:33.447274 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.447249    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-67fqk"]
Jul 29 14:35:33.447404 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.447377    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bxjbs"
Jul 29 14:35:33.450980 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.450962    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 29 14:35:33.451520 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.451409    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 29 14:35:33.452042 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.452022    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 29 14:35:33.452436 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.452317    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 29 14:35:33.462805 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.462780    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-tkzpb\""
Jul 29 14:35:33.462959 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.462939    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-lbmwl"]
Jul 29 14:35:33.463029 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.463015    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-67fqk"
Jul 29 14:35:33.464817 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.464795    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 29 14:35:33.464934 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.464818    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 29 14:35:33.465014 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.464985    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 29 14:35:33.465174 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.465153    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-g6266\""
Jul 29 14:35:33.482406 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.482379    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9j4jg"]
Jul 29 14:35:33.482533 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.482460    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvbtr"
Jul 29 14:35:33.482533 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.482476    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-lbmwl"
Jul 29 14:35:33.482760 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.482743    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vg8f5"
Jul 29 14:35:33.482928 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.482910    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vslsw"
Jul 29 14:35:33.484576 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.484555    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 14:35:33.484796 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.484776    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 14:35:33.484902 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.484866    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-cztgk\""
Jul 29 14:35:33.484969 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.484779    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 14:35:33.485025 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.485013    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ht8tz\""
Jul 29 14:35:33.485079 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.485063    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 14:35:33.485321 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.485301    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nzccd\""
Jul 29 14:35:33.500697 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.500677    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4m4xj"]
Jul 29 14:35:33.500837 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.500816    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9j4jg"
Jul 29 14:35:33.503491 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.503438    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 29 14:35:33.503652 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.503616    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 29 14:35:33.503766 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.503652    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-lm2w8\""
Jul 29 14:35:33.503855 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.503837    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 29 14:35:33.504175 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.503989    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 29 14:35:33.506159 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.506124    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/23d79588-937d-4c6d-8760-a081024f608e-trusted-ca\") pod \"console-operator-584dcb6b54-xnm95\" (UID: \"23d79588-937d-4c6d-8760-a081024f608e\") " pod="openshift-console-operator/console-operator-584dcb6b54-xnm95"
Jul 29 14:35:33.506322 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.506294    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-image-registry-private-configuration\") pod \"image-registry-5d8957c4d5-4pb5r\" (UID: \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\") " pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r"
Jul 29 14:35:33.506404 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.506332    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-registry-tls\") pod \"image-registry-5d8957c4d5-4pb5r\" (UID: \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\") " pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r"
Jul 29 14:35:33.506404 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.506357    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-ca-trust-extracted\") pod \"image-registry-5d8957c4d5-4pb5r\" (UID: \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\") " pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r"
Jul 29 14:35:33.506509 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.506405    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65jr9\" (UniqueName: \"kubernetes.io/projected/c1eb675a-8bbd-48ef-a526-506a7fd20ee7-kube-api-access-65jr9\") pod \"volume-data-source-validator-549c6d54cf-g9dbl\" (UID: \"c1eb675a-8bbd-48ef-a526-506a7fd20ee7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-g9dbl"
Jul 29 14:35:33.506509 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.506430    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/23d79588-937d-4c6d-8760-a081024f608e-serving-cert\") pod \"console-operator-584dcb6b54-xnm95\" (UID: \"23d79588-937d-4c6d-8760-a081024f608e\") " pod="openshift-console-operator/console-operator-584dcb6b54-xnm95"
Jul 29 14:35:33.506509 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.506454    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-trusted-ca\") pod \"image-registry-5d8957c4d5-4pb5r\" (UID: \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\") " pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r"
Jul 29 14:35:33.506509 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.506486    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrjtl\" (UniqueName: \"kubernetes.io/projected/7504b97a-38e2-44ff-822a-a7f1e98b8933-kube-api-access-wrjtl\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-fkjfz\" (UID: \"7504b97a-38e2-44ff-822a-a7f1e98b8933\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-fkjfz"
Jul 29 14:35:33.506701 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.506541    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15611a65-9454-4396-9b2f-fc9599672fc4-service-ca-bundle\") pod \"insights-operator-5885fdb759-nh8pd\" (UID: \"15611a65-9454-4396-9b2f-fc9599672fc4\") " pod="openshift-insights/insights-operator-5885fdb759-nh8pd"
Jul 29 14:35:33.506701 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.506563    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/23d79588-937d-4c6d-8760-a081024f608e-config\") pod \"console-operator-584dcb6b54-xnm95\" (UID: \"23d79588-937d-4c6d-8760-a081024f608e\") " pod="openshift-console-operator/console-operator-584dcb6b54-xnm95"
Jul 29 14:35:33.506701 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.506587    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-installation-pull-secrets\") pod \"image-registry-5d8957c4d5-4pb5r\" (UID: \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\") " pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r"
Jul 29 14:35:33.506701 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.506615    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15611a65-9454-4396-9b2f-fc9599672fc4-tmp\") pod \"insights-operator-5885fdb759-nh8pd\" (UID: \"15611a65-9454-4396-9b2f-fc9599672fc4\") " pod="openshift-insights/insights-operator-5885fdb759-nh8pd"
Jul 29 14:35:33.506701 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.506641    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75dk6\" (UniqueName: \"kubernetes.io/projected/15611a65-9454-4396-9b2f-fc9599672fc4-kube-api-access-75dk6\") pod \"insights-operator-5885fdb759-nh8pd\" (UID: \"15611a65-9454-4396-9b2f-fc9599672fc4\") " pod="openshift-insights/insights-operator-5885fdb759-nh8pd"
Jul 29 14:35:33.506701 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.506667    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqdqk\" (UniqueName: \"kubernetes.io/projected/23d79588-937d-4c6d-8760-a081024f608e-kube-api-access-hqdqk\") pod \"console-operator-584dcb6b54-xnm95\" (UID: \"23d79588-937d-4c6d-8760-a081024f608e\") " pod="openshift-console-operator/console-operator-584dcb6b54-xnm95"
Jul 29 14:35:33.506701 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.506692    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-registry-certificates\") pod \"image-registry-5d8957c4d5-4pb5r\" (UID: \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\") " pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r"
Jul 29 14:35:33.507015 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.506718    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/15611a65-9454-4396-9b2f-fc9599672fc4-snapshots\") pod \"insights-operator-5885fdb759-nh8pd\" (UID: \"15611a65-9454-4396-9b2f-fc9599672fc4\") " pod="openshift-insights/insights-operator-5885fdb759-nh8pd"
Jul 29 14:35:33.507015 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.506749    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7504b97a-38e2-44ff-822a-a7f1e98b8933-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-fkjfz\" (UID: \"7504b97a-38e2-44ff-822a-a7f1e98b8933\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-fkjfz"
Jul 29 14:35:33.507015 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.506787    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7504b97a-38e2-44ff-822a-a7f1e98b8933-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-fkjfz\" (UID: \"7504b97a-38e2-44ff-822a-a7f1e98b8933\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-fkjfz"
Jul 29 14:35:33.507015 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.506830    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsccx\" (UniqueName: \"kubernetes.io/projected/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-kube-api-access-jsccx\") pod \"image-registry-5d8957c4d5-4pb5r\" (UID: \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\") " pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r"
Jul 29 14:35:33.507015 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.506868    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/15611a65-9454-4396-9b2f-fc9599672fc4-serving-cert\") pod \"insights-operator-5885fdb759-nh8pd\" (UID: \"15611a65-9454-4396-9b2f-fc9599672fc4\") " pod="openshift-insights/insights-operator-5885fdb759-nh8pd"
Jul 29 14:35:33.507015 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.506900    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15611a65-9454-4396-9b2f-fc9599672fc4-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-nh8pd\" (UID: \"15611a65-9454-4396-9b2f-fc9599672fc4\") " pod="openshift-insights/insights-operator-5885fdb759-nh8pd"
Jul 29 14:35:33.507015 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.506923    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-bound-sa-token\") pod \"image-registry-5d8957c4d5-4pb5r\" (UID: \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\") " pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r"
Jul 29 14:35:33.518568 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.518549    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65jr9\" (UniqueName: \"kubernetes.io/projected/c1eb675a-8bbd-48ef-a526-506a7fd20ee7-kube-api-access-65jr9\") pod \"volume-data-source-validator-549c6d54cf-g9dbl\" (UID: \"c1eb675a-8bbd-48ef-a526-506a7fd20ee7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-g9dbl"
Jul 29 14:35:33.519051 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.518895    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-lbmwl"]
Jul 29 14:35:33.519051 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.518935    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qcjfr"]
Jul 29 14:35:33.519336 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.519318    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4m4xj"
Jul 29 14:35:33.521445 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.521426    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pp4lb\""
Jul 29 14:35:33.521538 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.521430    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 29 14:35:33.521657 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.521462    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 29 14:35:33.536970 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.536944    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9j4jg"]
Jul 29 14:35:33.537105 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.536985    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-fkjfz"]
Jul 29 14:35:33.537105 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.537028    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-67fqk"]
Jul 29 14:35:33.537105 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.537049    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-746f69dff9-2bdj7"]
Jul 29 14:35:33.537105 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.537069    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-bxjbs"]
Jul 29 14:35:33.537105 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.537070    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qcjfr"
Jul 29 14:35:33.537105 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.537089    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qcjfr"]
Jul 29 14:35:33.537105 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.537102    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4m4xj"]
Jul 29 14:35:33.537515 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.537118    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-nh8pd"]
Jul 29 14:35:33.539333 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.539311    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 29 14:35:33.539438 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.539350    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fb2k5\""
Jul 29 14:35:33.539438 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.539399    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 29 14:35:33.539438 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.539427    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 29 14:35:33.607561 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.607484    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-bound-sa-token\") pod \"image-registry-5d8957c4d5-4pb5r\" (UID: \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\") " pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r"
Jul 29 14:35:33.607561 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.607526    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h286d\" (UniqueName: \"kubernetes.io/projected/f2037177-a695-47b3-bda0-0669bcfe408e-kube-api-access-h286d\") pod \"cluster-monitoring-operator-548b489cb8-bxjbs\" (UID: \"f2037177-a695-47b3-bda0-0669bcfe408e\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bxjbs"
Jul 29 14:35:33.607561 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.607555    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/23d79588-937d-4c6d-8760-a081024f608e-serving-cert\") pod \"console-operator-584dcb6b54-xnm95\" (UID: \"23d79588-937d-4c6d-8760-a081024f608e\") " pod="openshift-console-operator/console-operator-584dcb6b54-xnm95"
Jul 29 14:35:33.607824 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.607577    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cca0fc93-ab3f-4cb2-aaab-8fcdc4428636-serving-cert\") pod \"service-ca-operator-5d6c75f84f-9j4jg\" (UID: \"cca0fc93-ab3f-4cb2-aaab-8fcdc4428636\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9j4jg"
Jul 29 14:35:33.607824 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.607606    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4dbfdc61-952e-49ee-a2b2-a484d295b1e3-config-volume\") pod \"dns-default-4m4xj\" (UID: \"4dbfdc61-952e-49ee-a2b2-a484d295b1e3\") " pod="openshift-dns/dns-default-4m4xj"
Jul 29 14:35:33.607824 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.607635    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75dk6\" (UniqueName: \"kubernetes.io/projected/15611a65-9454-4396-9b2f-fc9599672fc4-kube-api-access-75dk6\") pod \"insights-operator-5885fdb759-nh8pd\" (UID: \"15611a65-9454-4396-9b2f-fc9599672fc4\") " pod="openshift-insights/insights-operator-5885fdb759-nh8pd"
Jul 29 14:35:33.607824 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.607666    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15611a65-9454-4396-9b2f-fc9599672fc4-service-ca-bundle\") pod \"insights-operator-5885fdb759-nh8pd\" (UID: \"15611a65-9454-4396-9b2f-fc9599672fc4\") " pod="openshift-insights/insights-operator-5885fdb759-nh8pd"
Jul 29 14:35:33.607824 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.607705    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/23d79588-937d-4c6d-8760-a081024f608e-config\") pod \"console-operator-584dcb6b54-xnm95\" (UID: \"23d79588-937d-4c6d-8760-a081024f608e\") " pod="openshift-console-operator/console-operator-584dcb6b54-xnm95"
Jul 29 14:35:33.607824 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.607729    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15611a65-9454-4396-9b2f-fc9599672fc4-tmp\") pod \"insights-operator-5885fdb759-nh8pd\" (UID: \"15611a65-9454-4396-9b2f-fc9599672fc4\") " pod="openshift-insights/insights-operator-5885fdb759-nh8pd"
Jul 29 14:35:33.607824 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.607750    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2037177-a695-47b3-bda0-0669bcfe408e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-bxjbs\" (UID: \"f2037177-a695-47b3-bda0-0669bcfe408e\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bxjbs"
Jul 29 14:35:33.607824 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.607776    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f2037177-a695-47b3-bda0-0669bcfe408e-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-bxjbs\" (UID: \"f2037177-a695-47b3-bda0-0669bcfe408e\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bxjbs"
Jul 29 14:35:33.607824 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.607801    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f010d9fc-7827-43c6-b9d8-03b9b09222ad-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-67fqk\" (UID: \"f010d9fc-7827-43c6-b9d8-03b9b09222ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-67fqk"
Jul 29 14:35:33.607824 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.607826    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpvk2\" (UniqueName: \"kubernetes.io/projected/cca0fc93-ab3f-4cb2-aaab-8fcdc4428636-kube-api-access-jpvk2\") pod \"service-ca-operator-5d6c75f84f-9j4jg\" (UID: \"cca0fc93-ab3f-4cb2-aaab-8fcdc4428636\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9j4jg"
Jul 29 14:35:33.608329 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.607868    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsccx\" (UniqueName: \"kubernetes.io/projected/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-kube-api-access-jsccx\") pod \"image-registry-5d8957c4d5-4pb5r\" (UID: \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\") " pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r"
Jul 29 14:35:33.608329 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.607911    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/15611a65-9454-4396-9b2f-fc9599672fc4-serving-cert\") pod \"insights-operator-5885fdb759-nh8pd\" (UID: \"15611a65-9454-4396-9b2f-fc9599672fc4\") " pod="openshift-insights/insights-operator-5885fdb759-nh8pd"
Jul 29 14:35:33.608329 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.607940    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15611a65-9454-4396-9b2f-fc9599672fc4-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-nh8pd\" (UID: \"15611a65-9454-4396-9b2f-fc9599672fc4\") " pod="openshift-insights/insights-operator-5885fdb759-nh8pd"
Jul 29 14:35:33.608329 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.607962    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-stats-auth\") pod \"router-default-746f69dff9-2bdj7\" (UID: \"1a5f8182-95a6-4f4b-b40b-ea003c7be4dd\") " pod="openshift-ingress/router-default-746f69dff9-2bdj7"
Jul 29 14:35:33.608329 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.607988    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/23d79588-937d-4c6d-8760-a081024f608e-trusted-ca\") pod \"console-operator-584dcb6b54-xnm95\" (UID: \"23d79588-937d-4c6d-8760-a081024f608e\") " pod="openshift-console-operator/console-operator-584dcb6b54-xnm95"
Jul 29 14:35:33.608329 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.608011    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-image-registry-private-configuration\") pod \"image-registry-5d8957c4d5-4pb5r\" (UID: \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\") " pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r"
Jul 29 14:35:33.608329 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.608034    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-ca-trust-extracted\") pod \"image-registry-5d8957c4d5-4pb5r\" (UID: \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\") " pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r"
Jul 29 14:35:33.608329 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.608060    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrjtl\" (UniqueName: \"kubernetes.io/projected/7504b97a-38e2-44ff-822a-a7f1e98b8933-kube-api-access-wrjtl\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-fkjfz\" (UID: \"7504b97a-38e2-44ff-822a-a7f1e98b8933\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-fkjfz"
Jul 29 14:35:33.608329 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.608089    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4dbfdc61-952e-49ee-a2b2-a484d295b1e3-metrics-tls\") pod \"dns-default-4m4xj\" (UID: \"4dbfdc61-952e-49ee-a2b2-a484d295b1e3\") " pod="openshift-dns/dns-default-4m4xj"
Jul 29 14:35:33.608329 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.608116    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-registry-tls\") pod \"image-registry-5d8957c4d5-4pb5r\" (UID: \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\") " pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r"
Jul 29 14:35:33.608329 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.608153    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-trusted-ca\") pod \"image-registry-5d8957c4d5-4pb5r\" (UID: \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\") " pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r"
Jul 29 14:35:33.608329 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.608177    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dbfdc61-952e-49ee-a2b2-a484d295b1e3-tmp-dir\") pod \"dns-default-4m4xj\" (UID: \"4dbfdc61-952e-49ee-a2b2-a484d295b1e3\") " pod="openshift-dns/dns-default-4m4xj"
Jul 29 14:35:33.608862 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.608577    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15611a65-9454-4396-9b2f-fc9599672fc4-service-ca-bundle\") pod \"insights-operator-5885fdb759-nh8pd\" (UID: \"15611a65-9454-4396-9b2f-fc9599672fc4\") " pod="openshift-insights/insights-operator-5885fdb759-nh8pd"
Jul 29 14:35:33.608862 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.608647    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/23d79588-937d-4c6d-8760-a081024f608e-config\") pod \"console-operator-584dcb6b54-xnm95\" (UID: \"23d79588-937d-4c6d-8760-a081024f608e\") " pod="openshift-console-operator/console-operator-584dcb6b54-xnm95"
Jul 29 14:35:33.608862 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.608704    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kf25\" (UniqueName: \"kubernetes.io/projected/4dbfdc61-952e-49ee-a2b2-a484d295b1e3-kube-api-access-4kf25\") pod \"dns-default-4m4xj\" (UID: \"4dbfdc61-952e-49ee-a2b2-a484d295b1e3\") " pod="openshift-dns/dns-default-4m4xj"
Jul 29 14:35:33.608862 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.608735    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgmk2\" (UniqueName: \"kubernetes.io/projected/5f530514-b6e9-4e5e-9f31-4d6a0792f742-kube-api-access-mgmk2\") pod \"network-check-source-7d88cb55d8-lbmwl\" (UID: \"5f530514-b6e9-4e5e-9f31-4d6a0792f742\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-lbmwl"
Jul 29 14:35:33.608862 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.608786    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-default-certificate\") pod \"router-default-746f69dff9-2bdj7\" (UID: \"1a5f8182-95a6-4f4b-b40b-ea003c7be4dd\") " pod="openshift-ingress/router-default-746f69dff9-2bdj7"
Jul 29 14:35:33.609091 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:33.608864    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 14:35:33.609091 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:33.608877    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d8957c4d5-4pb5r: secret "image-registry-tls" not found
Jul 29 14:35:33.609091 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:33.608928    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-registry-tls podName:51a472c7-19c4-4c5c-9813-ebf6cb7dd70c nodeName:}" failed. No retries permitted until 2026-07-29 14:35:34.108911623 +0000 UTC m=+36.157369545 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-registry-tls") pod "image-registry-5d8957c4d5-4pb5r" (UID: "51a472c7-19c4-4c5c-9813-ebf6cb7dd70c") : secret "image-registry-tls" not found
Jul 29 14:35:33.609281 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.609091    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15611a65-9454-4396-9b2f-fc9599672fc4-tmp\") pod \"insights-operator-5885fdb759-nh8pd\" (UID: \"15611a65-9454-4396-9b2f-fc9599672fc4\") " pod="openshift-insights/insights-operator-5885fdb759-nh8pd"
Jul 29 14:35:33.609281 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.609109    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/23d79588-937d-4c6d-8760-a081024f608e-trusted-ca\") pod \"console-operator-584dcb6b54-xnm95\" (UID: \"23d79588-937d-4c6d-8760-a081024f608e\") " pod="openshift-console-operator/console-operator-584dcb6b54-xnm95"
Jul 29 14:35:33.609281 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.609250    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-trusted-ca\") pod \"image-registry-5d8957c4d5-4pb5r\" (UID: \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\") " pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r"
Jul 29 14:35:33.609281 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.609255    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-installation-pull-secrets\") pod \"image-registry-5d8957c4d5-4pb5r\" (UID: \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\") " pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r"
Jul 29 14:35:33.609479 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.609303    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stgxj\" (UniqueName: \"kubernetes.io/projected/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-kube-api-access-stgxj\") pod \"router-default-746f69dff9-2bdj7\" (UID: \"1a5f8182-95a6-4f4b-b40b-ea003c7be4dd\") " pod="openshift-ingress/router-default-746f69dff9-2bdj7"
Jul 29 14:35:33.609479 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.609424    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-ca-trust-extracted\") pod \"image-registry-5d8957c4d5-4pb5r\" (UID: \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\") " pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r"
Jul 29 14:35:33.609479 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.609453    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqdqk\" (UniqueName: \"kubernetes.io/projected/23d79588-937d-4c6d-8760-a081024f608e-kube-api-access-hqdqk\") pod \"console-operator-584dcb6b54-xnm95\" (UID: \"23d79588-937d-4c6d-8760-a081024f608e\") " pod="openshift-console-operator/console-operator-584dcb6b54-xnm95"
Jul 29 14:35:33.609625 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.609491    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-metrics-certs\") pod \"router-default-746f69dff9-2bdj7\" (UID: \"1a5f8182-95a6-4f4b-b40b-ea003c7be4dd\") " pod="openshift-ingress/router-default-746f69dff9-2bdj7"
Jul 29 14:35:33.609625 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.609549    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15611a65-9454-4396-9b2f-fc9599672fc4-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-nh8pd\" (UID: \"15611a65-9454-4396-9b2f-fc9599672fc4\") " pod="openshift-insights/insights-operator-5885fdb759-nh8pd"
Jul 29 14:35:33.609625 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.609581    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-registry-certificates\") pod \"image-registry-5d8957c4d5-4pb5r\" (UID: \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\") " pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r"
Jul 29 14:35:33.609767 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.609624    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-service-ca-bundle\") pod \"router-default-746f69dff9-2bdj7\" (UID: \"1a5f8182-95a6-4f4b-b40b-ea003c7be4dd\") " pod="openshift-ingress/router-default-746f69dff9-2bdj7"
Jul 29 14:35:33.609767 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.609652    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/15611a65-9454-4396-9b2f-fc9599672fc4-snapshots\") pod \"insights-operator-5885fdb759-nh8pd\" (UID: \"15611a65-9454-4396-9b2f-fc9599672fc4\") " pod="openshift-insights/insights-operator-5885fdb759-nh8pd"
Jul 29 14:35:33.609767 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.609677    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cca0fc93-ab3f-4cb2-aaab-8fcdc4428636-config\") pod \"service-ca-operator-5d6c75f84f-9j4jg\" (UID: \"cca0fc93-ab3f-4cb2-aaab-8fcdc4428636\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9j4jg"
Jul 29 14:35:33.609767 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.609709    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7504b97a-38e2-44ff-822a-a7f1e98b8933-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-fkjfz\" (UID: \"7504b97a-38e2-44ff-822a-a7f1e98b8933\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-fkjfz"
Jul 29 14:35:33.609767 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.609733    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7504b97a-38e2-44ff-822a-a7f1e98b8933-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-fkjfz\" (UID: \"7504b97a-38e2-44ff-822a-a7f1e98b8933\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-fkjfz"
Jul 29 14:35:33.609767 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.609760    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g89xt\" (UniqueName: \"kubernetes.io/projected/f010d9fc-7827-43c6-b9d8-03b9b09222ad-kube-api-access-g89xt\") pod \"cluster-samples-operator-6748489656-67fqk\" (UID: \"f010d9fc-7827-43c6-b9d8-03b9b09222ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-67fqk"
Jul 29 14:35:33.610323 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.610283    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/15611a65-9454-4396-9b2f-fc9599672fc4-snapshots\") pod \"insights-operator-5885fdb759-nh8pd\" (UID: \"15611a65-9454-4396-9b2f-fc9599672fc4\") " pod="openshift-insights/insights-operator-5885fdb759-nh8pd"
Jul 29 14:35:33.610654 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.610613    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7504b97a-38e2-44ff-822a-a7f1e98b8933-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-fkjfz\" (UID: \"7504b97a-38e2-44ff-822a-a7f1e98b8933\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-fkjfz"
Jul 29 14:35:33.610829 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.610808    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-registry-certificates\") pod \"image-registry-5d8957c4d5-4pb5r\" (UID: \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\") " pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r"
Jul 29 14:35:33.611076 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.611051    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/23d79588-937d-4c6d-8760-a081024f608e-serving-cert\") pod \"console-operator-584dcb6b54-xnm95\" (UID: \"23d79588-937d-4c6d-8760-a081024f608e\") " pod="openshift-console-operator/console-operator-584dcb6b54-xnm95"
Jul 29 14:35:33.611658 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.611637    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-image-registry-private-configuration\") pod \"image-registry-5d8957c4d5-4pb5r\" (UID: \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\") " pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r"
Jul 29 14:35:33.611918 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.611897    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/15611a65-9454-4396-9b2f-fc9599672fc4-serving-cert\") pod \"insights-operator-5885fdb759-nh8pd\" (UID: \"15611a65-9454-4396-9b2f-fc9599672fc4\") " pod="openshift-insights/insights-operator-5885fdb759-nh8pd"
Jul 29 14:35:33.612328 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.612300    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7504b97a-38e2-44ff-822a-a7f1e98b8933-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-fkjfz\" (UID: \"7504b97a-38e2-44ff-822a-a7f1e98b8933\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-fkjfz"
Jul 29 14:35:33.612871 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.612849    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-installation-pull-secrets\") pod \"image-registry-5d8957c4d5-4pb5r\" (UID: \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\") " pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r"
Jul 29 14:35:33.618999 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.618975    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-bound-sa-token\") pod \"image-registry-5d8957c4d5-4pb5r\" (UID: \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\") " pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r"
Jul 29 14:35:33.619555 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.619532    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqdqk\" (UniqueName: \"kubernetes.io/projected/23d79588-937d-4c6d-8760-a081024f608e-kube-api-access-hqdqk\") pod \"console-operator-584dcb6b54-xnm95\" (UID: \"23d79588-937d-4c6d-8760-a081024f608e\") " pod="openshift-console-operator/console-operator-584dcb6b54-xnm95"
Jul 29 14:35:33.619801 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.619779    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrjtl\" (UniqueName: \"kubernetes.io/projected/7504b97a-38e2-44ff-822a-a7f1e98b8933-kube-api-access-wrjtl\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-fkjfz\" (UID: \"7504b97a-38e2-44ff-822a-a7f1e98b8933\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-fkjfz"
Jul 29 14:35:33.620012 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.619994    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsccx\" (UniqueName: \"kubernetes.io/projected/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-kube-api-access-jsccx\") pod \"image-registry-5d8957c4d5-4pb5r\" (UID: \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\") " pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r"
Jul 29 14:35:33.621287 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.621265    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75dk6\" (UniqueName: \"kubernetes.io/projected/15611a65-9454-4396-9b2f-fc9599672fc4-kube-api-access-75dk6\") pod \"insights-operator-5885fdb759-nh8pd\" (UID: \"15611a65-9454-4396-9b2f-fc9599672fc4\") " pod="openshift-insights/insights-operator-5885fdb759-nh8pd"
Jul 29 14:35:33.631054 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.631025    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-g9dbl"
Jul 29 14:35:33.659620 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.659596    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-xnm95"
Jul 29 14:35:33.694236 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.694199    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-nh8pd"
Jul 29 14:35:33.710666 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.710629    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h286d\" (UniqueName: \"kubernetes.io/projected/f2037177-a695-47b3-bda0-0669bcfe408e-kube-api-access-h286d\") pod \"cluster-monitoring-operator-548b489cb8-bxjbs\" (UID: \"f2037177-a695-47b3-bda0-0669bcfe408e\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bxjbs"
Jul 29 14:35:33.710850 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.710687    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cca0fc93-ab3f-4cb2-aaab-8fcdc4428636-serving-cert\") pod \"service-ca-operator-5d6c75f84f-9j4jg\" (UID: \"cca0fc93-ab3f-4cb2-aaab-8fcdc4428636\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9j4jg"
Jul 29 14:35:33.710850 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.710731    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4dbfdc61-952e-49ee-a2b2-a484d295b1e3-config-volume\") pod \"dns-default-4m4xj\" (UID: \"4dbfdc61-952e-49ee-a2b2-a484d295b1e3\") " pod="openshift-dns/dns-default-4m4xj"
Jul 29 14:35:33.710850 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.710771    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2037177-a695-47b3-bda0-0669bcfe408e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-bxjbs\" (UID: \"f2037177-a695-47b3-bda0-0669bcfe408e\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bxjbs"
Jul 29 14:35:33.710850 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.710800    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f2037177-a695-47b3-bda0-0669bcfe408e-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-bxjbs\" (UID: \"f2037177-a695-47b3-bda0-0669bcfe408e\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bxjbs"
Jul 29 14:35:33.710850 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.710824    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f010d9fc-7827-43c6-b9d8-03b9b09222ad-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-67fqk\" (UID: \"f010d9fc-7827-43c6-b9d8-03b9b09222ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-67fqk"
Jul 29 14:35:33.711813 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.710873    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpvk2\" (UniqueName: \"kubernetes.io/projected/cca0fc93-ab3f-4cb2-aaab-8fcdc4428636-kube-api-access-jpvk2\") pod \"service-ca-operator-5d6c75f84f-9j4jg\" (UID: \"cca0fc93-ab3f-4cb2-aaab-8fcdc4428636\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9j4jg"
Jul 29 14:35:33.711813 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.710959    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-stats-auth\") pod \"router-default-746f69dff9-2bdj7\" (UID: \"1a5f8182-95a6-4f4b-b40b-ea003c7be4dd\") " pod="openshift-ingress/router-default-746f69dff9-2bdj7"
Jul 29 14:35:33.711813 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.710987    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4dbfdc61-952e-49ee-a2b2-a484d295b1e3-metrics-tls\") pod \"dns-default-4m4xj\" (UID: \"4dbfdc61-952e-49ee-a2b2-a484d295b1e3\") " pod="openshift-dns/dns-default-4m4xj"
Jul 29 14:35:33.711813 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.711029    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dbfdc61-952e-49ee-a2b2-a484d295b1e3-tmp-dir\") pod \"dns-default-4m4xj\" (UID: \"4dbfdc61-952e-49ee-a2b2-a484d295b1e3\") " pod="openshift-dns/dns-default-4m4xj"
Jul 29 14:35:33.711813 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:33.711053    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 14:35:33.711813 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:33.711124    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f010d9fc-7827-43c6-b9d8-03b9b09222ad-samples-operator-tls podName:f010d9fc-7827-43c6-b9d8-03b9b09222ad nodeName:}" failed. No retries permitted until 2026-07-29 14:35:34.21110173 +0000 UTC m=+36.259559671 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f010d9fc-7827-43c6-b9d8-03b9b09222ad-samples-operator-tls") pod "cluster-samples-operator-6748489656-67fqk" (UID: "f010d9fc-7827-43c6-b9d8-03b9b09222ad") : secret "samples-operator-tls" not found
Jul 29 14:35:33.711813 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:33.711312    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 14:35:33.711813 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.711054    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kf25\" (UniqueName: \"kubernetes.io/projected/4dbfdc61-952e-49ee-a2b2-a484d295b1e3-kube-api-access-4kf25\") pod \"dns-default-4m4xj\" (UID: \"4dbfdc61-952e-49ee-a2b2-a484d295b1e3\") " pod="openshift-dns/dns-default-4m4xj"
Jul 29 14:35:33.711813 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:33.711353    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dbfdc61-952e-49ee-a2b2-a484d295b1e3-metrics-tls podName:4dbfdc61-952e-49ee-a2b2-a484d295b1e3 nodeName:}" failed. No retries permitted until 2026-07-29 14:35:34.211340371 +0000 UTC m=+36.259798324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4dbfdc61-952e-49ee-a2b2-a484d295b1e3-metrics-tls") pod "dns-default-4m4xj" (UID: "4dbfdc61-952e-49ee-a2b2-a484d295b1e3") : secret "dns-default-metrics-tls" not found
Jul 29 14:35:33.711813 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:33.711424    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 14:35:33.711813 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.711449    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4dbfdc61-952e-49ee-a2b2-a484d295b1e3-config-volume\") pod \"dns-default-4m4xj\" (UID: \"4dbfdc61-952e-49ee-a2b2-a484d295b1e3\") " pod="openshift-dns/dns-default-4m4xj"
Jul 29 14:35:33.711813 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:33.711468    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2037177-a695-47b3-bda0-0669bcfe408e-cluster-monitoring-operator-tls podName:f2037177-a695-47b3-bda0-0669bcfe408e nodeName:}" failed. No retries permitted until 2026-07-29 14:35:34.21145429 +0000 UTC m=+36.259912209 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f2037177-a695-47b3-bda0-0669bcfe408e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-bxjbs" (UID: "f2037177-a695-47b3-bda0-0669bcfe408e") : secret "cluster-monitoring-operator-tls" not found
Jul 29 14:35:33.711813 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.711495    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgmk2\" (UniqueName: \"kubernetes.io/projected/5f530514-b6e9-4e5e-9f31-4d6a0792f742-kube-api-access-mgmk2\") pod \"network-check-source-7d88cb55d8-lbmwl\" (UID: \"5f530514-b6e9-4e5e-9f31-4d6a0792f742\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-lbmwl"
Jul 29 14:35:33.711813 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.711557    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-default-certificate\") pod \"router-default-746f69dff9-2bdj7\" (UID: \"1a5f8182-95a6-4f4b-b40b-ea003c7be4dd\") " pod="openshift-ingress/router-default-746f69dff9-2bdj7"
Jul 29 14:35:33.711813 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.711592    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/031b9fd3-1851-451a-b5e4-7f667c641263-cert\") pod \"ingress-canary-qcjfr\" (UID: \"031b9fd3-1851-451a-b5e4-7f667c641263\") " pod="openshift-ingress-canary/ingress-canary-qcjfr"
Jul 29 14:35:33.711813 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.711668    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stgxj\" (UniqueName: \"kubernetes.io/projected/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-kube-api-access-stgxj\") pod \"router-default-746f69dff9-2bdj7\" (UID: \"1a5f8182-95a6-4f4b-b40b-ea003c7be4dd\") " pod="openshift-ingress/router-default-746f69dff9-2bdj7"
Jul 29 14:35:33.712746 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.711700    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmccb\" (UniqueName: \"kubernetes.io/projected/031b9fd3-1851-451a-b5e4-7f667c641263-kube-api-access-dmccb\") pod \"ingress-canary-qcjfr\" (UID: \"031b9fd3-1851-451a-b5e4-7f667c641263\") " pod="openshift-ingress-canary/ingress-canary-qcjfr"
Jul 29 14:35:33.712746 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.711732    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-metrics-certs\") pod \"router-default-746f69dff9-2bdj7\" (UID: \"1a5f8182-95a6-4f4b-b40b-ea003c7be4dd\") " pod="openshift-ingress/router-default-746f69dff9-2bdj7"
Jul 29 14:35:33.712746 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.711766    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-service-ca-bundle\") pod \"router-default-746f69dff9-2bdj7\" (UID: \"1a5f8182-95a6-4f4b-b40b-ea003c7be4dd\") " pod="openshift-ingress/router-default-746f69dff9-2bdj7"
Jul 29 14:35:33.712746 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.711796    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dbfdc61-952e-49ee-a2b2-a484d295b1e3-tmp-dir\") pod \"dns-default-4m4xj\" (UID: \"4dbfdc61-952e-49ee-a2b2-a484d295b1e3\") " pod="openshift-dns/dns-default-4m4xj"
Jul 29 14:35:33.712746 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:33.711863    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-service-ca-bundle podName:1a5f8182-95a6-4f4b-b40b-ea003c7be4dd nodeName:}" failed. No retries permitted until 2026-07-29 14:35:34.21184918 +0000 UTC m=+36.260307099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-service-ca-bundle") pod "router-default-746f69dff9-2bdj7" (UID: "1a5f8182-95a6-4f4b-b40b-ea003c7be4dd") : configmap references non-existent config key: service-ca.crt
Jul 29 14:35:33.712746 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:33.711953    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 29 14:35:33.712746 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:33.711989    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-metrics-certs podName:1a5f8182-95a6-4f4b-b40b-ea003c7be4dd nodeName:}" failed. No retries permitted until 2026-07-29 14:35:34.211978538 +0000 UTC m=+36.260436473 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-metrics-certs") pod "router-default-746f69dff9-2bdj7" (UID: "1a5f8182-95a6-4f4b-b40b-ea003c7be4dd") : secret "router-metrics-certs-default" not found
Jul 29 14:35:33.712746 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.711986    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cca0fc93-ab3f-4cb2-aaab-8fcdc4428636-config\") pod \"service-ca-operator-5d6c75f84f-9j4jg\" (UID: \"cca0fc93-ab3f-4cb2-aaab-8fcdc4428636\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9j4jg"
Jul 29 14:35:33.712746 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.712000    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f2037177-a695-47b3-bda0-0669bcfe408e-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-bxjbs\" (UID: \"f2037177-a695-47b3-bda0-0669bcfe408e\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bxjbs"
Jul 29 14:35:33.712746 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.712029    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g89xt\" (UniqueName: \"kubernetes.io/projected/f010d9fc-7827-43c6-b9d8-03b9b09222ad-kube-api-access-g89xt\") pod \"cluster-samples-operator-6748489656-67fqk\" (UID: \"f010d9fc-7827-43c6-b9d8-03b9b09222ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-67fqk"
Jul 29 14:35:33.712746 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.712531    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cca0fc93-ab3f-4cb2-aaab-8fcdc4428636-config\") pod \"service-ca-operator-5d6c75f84f-9j4jg\" (UID: \"cca0fc93-ab3f-4cb2-aaab-8fcdc4428636\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9j4jg"
Jul 29 14:35:33.714199 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.714176    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-default-certificate\") pod \"router-default-746f69dff9-2bdj7\" (UID: \"1a5f8182-95a6-4f4b-b40b-ea003c7be4dd\") " pod="openshift-ingress/router-default-746f69dff9-2bdj7"
Jul 29 14:35:33.714281 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.714252    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-stats-auth\") pod \"router-default-746f69dff9-2bdj7\" (UID: \"1a5f8182-95a6-4f4b-b40b-ea003c7be4dd\") " pod="openshift-ingress/router-default-746f69dff9-2bdj7"
Jul 29 14:35:33.714875 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.714852    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cca0fc93-ab3f-4cb2-aaab-8fcdc4428636-serving-cert\") pod \"service-ca-operator-5d6c75f84f-9j4jg\" (UID: \"cca0fc93-ab3f-4cb2-aaab-8fcdc4428636\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9j4jg"
Jul 29 14:35:33.714970 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.714934    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-fkjfz"
Jul 29 14:35:33.720874 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.720820    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kf25\" (UniqueName: \"kubernetes.io/projected/4dbfdc61-952e-49ee-a2b2-a484d295b1e3-kube-api-access-4kf25\") pod \"dns-default-4m4xj\" (UID: \"4dbfdc61-952e-49ee-a2b2-a484d295b1e3\") " pod="openshift-dns/dns-default-4m4xj"
Jul 29 14:35:33.721881 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.721503    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h286d\" (UniqueName: \"kubernetes.io/projected/f2037177-a695-47b3-bda0-0669bcfe408e-kube-api-access-h286d\") pod \"cluster-monitoring-operator-548b489cb8-bxjbs\" (UID: \"f2037177-a695-47b3-bda0-0669bcfe408e\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bxjbs"
Jul 29 14:35:33.721881 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.721819    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpvk2\" (UniqueName: \"kubernetes.io/projected/cca0fc93-ab3f-4cb2-aaab-8fcdc4428636-kube-api-access-jpvk2\") pod \"service-ca-operator-5d6c75f84f-9j4jg\" (UID: \"cca0fc93-ab3f-4cb2-aaab-8fcdc4428636\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9j4jg"
Jul 29 14:35:33.722027 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.721883    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stgxj\" (UniqueName: \"kubernetes.io/projected/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-kube-api-access-stgxj\") pod \"router-default-746f69dff9-2bdj7\" (UID: \"1a5f8182-95a6-4f4b-b40b-ea003c7be4dd\") " pod="openshift-ingress/router-default-746f69dff9-2bdj7"
Jul 29 14:35:33.723601 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.723579    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgmk2\" (UniqueName: \"kubernetes.io/projected/5f530514-b6e9-4e5e-9f31-4d6a0792f742-kube-api-access-mgmk2\") pod \"network-check-source-7d88cb55d8-lbmwl\" (UID: \"5f530514-b6e9-4e5e-9f31-4d6a0792f742\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-lbmwl"
Jul 29 14:35:33.725147 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.724005    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g89xt\" (UniqueName: \"kubernetes.io/projected/f010d9fc-7827-43c6-b9d8-03b9b09222ad-kube-api-access-g89xt\") pod \"cluster-samples-operator-6748489656-67fqk\" (UID: \"f010d9fc-7827-43c6-b9d8-03b9b09222ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-67fqk"
Jul 29 14:35:33.809671 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.807389    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-lbmwl"
Jul 29 14:35:33.812628 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.812559    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/031b9fd3-1851-451a-b5e4-7f667c641263-cert\") pod \"ingress-canary-qcjfr\" (UID: \"031b9fd3-1851-451a-b5e4-7f667c641263\") " pod="openshift-ingress-canary/ingress-canary-qcjfr"
Jul 29 14:35:33.812789 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.812772    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmccb\" (UniqueName: \"kubernetes.io/projected/031b9fd3-1851-451a-b5e4-7f667c641263-kube-api-access-dmccb\") pod \"ingress-canary-qcjfr\" (UID: \"031b9fd3-1851-451a-b5e4-7f667c641263\") " pod="openshift-ingress-canary/ingress-canary-qcjfr"
Jul 29 14:35:33.821798 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.821707    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/031b9fd3-1851-451a-b5e4-7f667c641263-cert\") pod \"ingress-canary-qcjfr\" (UID: \"031b9fd3-1851-451a-b5e4-7f667c641263\") " pod="openshift-ingress-canary/ingress-canary-qcjfr"
Jul 29 14:35:33.826221 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.825905    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmccb\" (UniqueName: \"kubernetes.io/projected/031b9fd3-1851-451a-b5e4-7f667c641263-kube-api-access-dmccb\") pod \"ingress-canary-qcjfr\" (UID: \"031b9fd3-1851-451a-b5e4-7f667c641263\") " pod="openshift-ingress-canary/ingress-canary-qcjfr"
Jul 29 14:35:33.834621 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.832752    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9j4jg"
Jul 29 14:35:33.849818 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.849419    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qcjfr"
Jul 29 14:35:33.903155 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.900601    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-g9dbl"]
Jul 29 14:35:33.911100 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:35:33.911059    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1eb675a_8bbd_48ef_a526_506a7fd20ee7.slice/crio-70d0ac52744da8629220c9ca57e853a49ecb8f50a86691d6ea1b628f4eb6f9ca WatchSource:0}: Error finding container 70d0ac52744da8629220c9ca57e853a49ecb8f50a86691d6ea1b628f4eb6f9ca: Status 404 returned error can't find the container with id 70d0ac52744da8629220c9ca57e853a49ecb8f50a86691d6ea1b628f4eb6f9ca
Jul 29 14:35:33.912846 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.912508    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-xnm95"]
Jul 29 14:35:33.950183 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.950086    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-nh8pd"]
Jul 29 14:35:33.965218 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:35:33.965180    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7504b97a_38e2_44ff_822a_a7f1e98b8933.slice/crio-3bc8c01ef74465d1e2fa1cc9429daff644396ae37fab66ac1174bbe8dac56dac WatchSource:0}: Error finding container 3bc8c01ef74465d1e2fa1cc9429daff644396ae37fab66ac1174bbe8dac56dac: Status 404 returned error can't find the container with id 3bc8c01ef74465d1e2fa1cc9429daff644396ae37fab66ac1174bbe8dac56dac
Jul 29 14:35:33.966295 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:33.965993    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-fkjfz"]
Jul 29 14:35:33.967669 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:35:33.967536    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15611a65_9454_4396_9b2f_fc9599672fc4.slice/crio-6234d052a5ef64793221986ec987816aa4a15c69838436d53689f67b2f949e5b WatchSource:0}: Error finding container 6234d052a5ef64793221986ec987816aa4a15c69838436d53689f67b2f949e5b: Status 404 returned error can't find the container with id 6234d052a5ef64793221986ec987816aa4a15c69838436d53689f67b2f949e5b
Jul 29 14:35:34.026654 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:34.026580    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-lbmwl"]
Jul 29 14:35:34.031333 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:35:34.031306    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f530514_b6e9_4e5e_9f31_4d6a0792f742.slice/crio-bb46a0a685df46ac25250785fc574b2e93e24de3448957a2a4ed9e46a33cfe9f WatchSource:0}: Error finding container bb46a0a685df46ac25250785fc574b2e93e24de3448957a2a4ed9e46a33cfe9f: Status 404 returned error can't find the container with id bb46a0a685df46ac25250785fc574b2e93e24de3448957a2a4ed9e46a33cfe9f
Jul 29 14:35:34.048091 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:34.048062    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9j4jg"]
Jul 29 14:35:34.049235 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:34.049212    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qcjfr"]
Jul 29 14:35:34.051332 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:35:34.051296    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcca0fc93_ab3f_4cb2_aaab_8fcdc4428636.slice/crio-2676a2a0ac521b4d2031fedf30d28361bf5a1ea6dd8d9908fa55afc453f14653 WatchSource:0}: Error finding container 2676a2a0ac521b4d2031fedf30d28361bf5a1ea6dd8d9908fa55afc453f14653: Status 404 returned error can't find the container with id 2676a2a0ac521b4d2031fedf30d28361bf5a1ea6dd8d9908fa55afc453f14653
Jul 29 14:35:34.052396 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:35:34.052373    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod031b9fd3_1851_451a_b5e4_7f667c641263.slice/crio-3769ee8c3cbdf579d2af7c29c05fb640d1c4ae2aa08fbdaa1bd5e0dc826b4556 WatchSource:0}: Error finding container 3769ee8c3cbdf579d2af7c29c05fb640d1c4ae2aa08fbdaa1bd5e0dc826b4556: Status 404 returned error can't find the container with id 3769ee8c3cbdf579d2af7c29c05fb640d1c4ae2aa08fbdaa1bd5e0dc826b4556
Jul 29 14:35:34.115530 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:34.115456    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-registry-tls\") pod \"image-registry-5d8957c4d5-4pb5r\" (UID: \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\") " pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r"
Jul 29 14:35:34.115670 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:34.115623    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 14:35:34.115670 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:34.115646    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d8957c4d5-4pb5r: secret "image-registry-tls" not found
Jul 29 14:35:34.115763 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:34.115712    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-registry-tls podName:51a472c7-19c4-4c5c-9813-ebf6cb7dd70c nodeName:}" failed. No retries permitted until 2026-07-29 14:35:35.115694075 +0000 UTC m=+37.164152034 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-registry-tls") pod "image-registry-5d8957c4d5-4pb5r" (UID: "51a472c7-19c4-4c5c-9813-ebf6cb7dd70c") : secret "image-registry-tls" not found
Jul 29 14:35:34.216317 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:34.216284    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2037177-a695-47b3-bda0-0669bcfe408e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-bxjbs\" (UID: \"f2037177-a695-47b3-bda0-0669bcfe408e\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bxjbs"
Jul 29 14:35:34.216481 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:34.216328    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f010d9fc-7827-43c6-b9d8-03b9b09222ad-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-67fqk\" (UID: \"f010d9fc-7827-43c6-b9d8-03b9b09222ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-67fqk"
Jul 29 14:35:34.216481 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:34.216428    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 14:35:34.216583 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:34.216471    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4dbfdc61-952e-49ee-a2b2-a484d295b1e3-metrics-tls\") pod \"dns-default-4m4xj\" (UID: \"4dbfdc61-952e-49ee-a2b2-a484d295b1e3\") " pod="openshift-dns/dns-default-4m4xj"
Jul 29 14:35:34.216583 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:34.216495    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2037177-a695-47b3-bda0-0669bcfe408e-cluster-monitoring-operator-tls podName:f2037177-a695-47b3-bda0-0669bcfe408e nodeName:}" failed. No retries permitted until 2026-07-29 14:35:35.216479944 +0000 UTC m=+37.264937861 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f2037177-a695-47b3-bda0-0669bcfe408e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-bxjbs" (UID: "f2037177-a695-47b3-bda0-0669bcfe408e") : secret "cluster-monitoring-operator-tls" not found
Jul 29 14:35:34.216583 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:34.216437    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 14:35:34.216583 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:34.216536    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 14:35:34.216583 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:34.216570    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dbfdc61-952e-49ee-a2b2-a484d295b1e3-metrics-tls podName:4dbfdc61-952e-49ee-a2b2-a484d295b1e3 nodeName:}" failed. No retries permitted until 2026-07-29 14:35:35.216559783 +0000 UTC m=+37.265017700 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4dbfdc61-952e-49ee-a2b2-a484d295b1e3-metrics-tls") pod "dns-default-4m4xj" (UID: "4dbfdc61-952e-49ee-a2b2-a484d295b1e3") : secret "dns-default-metrics-tls" not found
Jul 29 14:35:34.216583 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:34.216567    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-metrics-certs\") pod \"router-default-746f69dff9-2bdj7\" (UID: \"1a5f8182-95a6-4f4b-b40b-ea003c7be4dd\") " pod="openshift-ingress/router-default-746f69dff9-2bdj7"
Jul 29 14:35:34.216583 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:34.216582    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f010d9fc-7827-43c6-b9d8-03b9b09222ad-samples-operator-tls podName:f010d9fc-7827-43c6-b9d8-03b9b09222ad nodeName:}" failed. No retries permitted until 2026-07-29 14:35:35.216576391 +0000 UTC m=+37.265034307 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f010d9fc-7827-43c6-b9d8-03b9b09222ad-samples-operator-tls") pod "cluster-samples-operator-6748489656-67fqk" (UID: "f010d9fc-7827-43c6-b9d8-03b9b09222ad") : secret "samples-operator-tls" not found
Jul 29 14:35:34.216830 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:34.216604    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-service-ca-bundle\") pod \"router-default-746f69dff9-2bdj7\" (UID: \"1a5f8182-95a6-4f4b-b40b-ea003c7be4dd\") " pod="openshift-ingress/router-default-746f69dff9-2bdj7"
Jul 29 14:35:34.216830 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:34.216611    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 29 14:35:34.216830 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:34.216645    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-metrics-certs podName:1a5f8182-95a6-4f4b-b40b-ea003c7be4dd nodeName:}" failed. No retries permitted until 2026-07-29 14:35:35.216636668 +0000 UTC m=+37.265094584 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-metrics-certs") pod "router-default-746f69dff9-2bdj7" (UID: "1a5f8182-95a6-4f4b-b40b-ea003c7be4dd") : secret "router-metrics-certs-default" not found
Jul 29 14:35:34.216830 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:34.216750    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-service-ca-bundle podName:1a5f8182-95a6-4f4b-b40b-ea003c7be4dd nodeName:}" failed. No retries permitted until 2026-07-29 14:35:35.216732876 +0000 UTC m=+37.265190804 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-service-ca-bundle") pod "router-default-746f69dff9-2bdj7" (UID: "1a5f8182-95a6-4f4b-b40b-ea003c7be4dd") : configmap references non-existent config key: service-ca.crt
Jul 29 14:35:34.638006 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:34.637975    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-g9dbl" event={"ID":"c1eb675a-8bbd-48ef-a526-506a7fd20ee7","Type":"ContainerStarted","Data":"70d0ac52744da8629220c9ca57e853a49ecb8f50a86691d6ea1b628f4eb6f9ca"}
Jul 29 14:35:34.639606 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:34.639578    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-lbmwl" event={"ID":"5f530514-b6e9-4e5e-9f31-4d6a0792f742","Type":"ContainerStarted","Data":"bb46a0a685df46ac25250785fc574b2e93e24de3448957a2a4ed9e46a33cfe9f"}
Jul 29 14:35:34.640792 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:34.640764    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-nh8pd" event={"ID":"15611a65-9454-4396-9b2f-fc9599672fc4","Type":"ContainerStarted","Data":"6234d052a5ef64793221986ec987816aa4a15c69838436d53689f67b2f949e5b"}
Jul 29 14:35:34.641992 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:34.641967    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9j4jg" event={"ID":"cca0fc93-ab3f-4cb2-aaab-8fcdc4428636","Type":"ContainerStarted","Data":"2676a2a0ac521b4d2031fedf30d28361bf5a1ea6dd8d9908fa55afc453f14653"}
Jul 29 14:35:34.643188 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:34.643160    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-fkjfz" event={"ID":"7504b97a-38e2-44ff-822a-a7f1e98b8933","Type":"ContainerStarted","Data":"3bc8c01ef74465d1e2fa1cc9429daff644396ae37fab66ac1174bbe8dac56dac"}
Jul 29 14:35:34.644373 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:34.644346    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qcjfr" event={"ID":"031b9fd3-1851-451a-b5e4-7f667c641263","Type":"ContainerStarted","Data":"3769ee8c3cbdf579d2af7c29c05fb640d1c4ae2aa08fbdaa1bd5e0dc826b4556"}
Jul 29 14:35:34.645502 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:34.645474    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-xnm95" event={"ID":"23d79588-937d-4c6d-8760-a081024f608e","Type":"ContainerStarted","Data":"783755d756412df702e8b3213d3635ef4281f7bb8f2c01ffa099c251366f129d"}
Jul 29 14:35:35.126167 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:35.126119    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-registry-tls\") pod \"image-registry-5d8957c4d5-4pb5r\" (UID: \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\") " pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r"
Jul 29 14:35:35.126344 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:35.126297    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 14:35:35.126344 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:35.126338    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d8957c4d5-4pb5r: secret "image-registry-tls" not found
Jul 29 14:35:35.126462 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:35.126404    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-registry-tls podName:51a472c7-19c4-4c5c-9813-ebf6cb7dd70c nodeName:}" failed. No retries permitted until 2026-07-29 14:35:37.12638464 +0000 UTC m=+39.174842557 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-registry-tls") pod "image-registry-5d8957c4d5-4pb5r" (UID: "51a472c7-19c4-4c5c-9813-ebf6cb7dd70c") : secret "image-registry-tls" not found
Jul 29 14:35:35.226835 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:35.226789    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2037177-a695-47b3-bda0-0669bcfe408e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-bxjbs\" (UID: \"f2037177-a695-47b3-bda0-0669bcfe408e\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bxjbs"
Jul 29 14:35:35.226835 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:35.226849    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f010d9fc-7827-43c6-b9d8-03b9b09222ad-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-67fqk\" (UID: \"f010d9fc-7827-43c6-b9d8-03b9b09222ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-67fqk"
Jul 29 14:35:35.227086 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:35.226898    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4dbfdc61-952e-49ee-a2b2-a484d295b1e3-metrics-tls\") pod \"dns-default-4m4xj\" (UID: \"4dbfdc61-952e-49ee-a2b2-a484d295b1e3\") " pod="openshift-dns/dns-default-4m4xj"
Jul 29 14:35:35.227086 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:35.226970    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-metrics-certs\") pod \"router-default-746f69dff9-2bdj7\" (UID: \"1a5f8182-95a6-4f4b-b40b-ea003c7be4dd\") " pod="openshift-ingress/router-default-746f69dff9-2bdj7"
Jul 29 14:35:35.227086 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:35.226976    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 14:35:35.227086 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:35.226997    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-service-ca-bundle\") pod \"router-default-746f69dff9-2bdj7\" (UID: \"1a5f8182-95a6-4f4b-b40b-ea003c7be4dd\") " pod="openshift-ingress/router-default-746f69dff9-2bdj7"
Jul 29 14:35:35.227086 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:35.227022    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 14:35:35.227086 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:35.227057    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2037177-a695-47b3-bda0-0669bcfe408e-cluster-monitoring-operator-tls podName:f2037177-a695-47b3-bda0-0669bcfe408e nodeName:}" failed. No retries permitted until 2026-07-29 14:35:37.227034052 +0000 UTC m=+39.275491970 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f2037177-a695-47b3-bda0-0669bcfe408e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-bxjbs" (UID: "f2037177-a695-47b3-bda0-0669bcfe408e") : secret "cluster-monitoring-operator-tls" not found
Jul 29 14:35:35.227086 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:35.227084    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f010d9fc-7827-43c6-b9d8-03b9b09222ad-samples-operator-tls podName:f010d9fc-7827-43c6-b9d8-03b9b09222ad nodeName:}" failed. No retries permitted until 2026-07-29 14:35:37.227068198 +0000 UTC m=+39.275526128 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f010d9fc-7827-43c6-b9d8-03b9b09222ad-samples-operator-tls") pod "cluster-samples-operator-6748489656-67fqk" (UID: "f010d9fc-7827-43c6-b9d8-03b9b09222ad") : secret "samples-operator-tls" not found
Jul 29 14:35:35.227086 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:35.227083    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 14:35:35.227483 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:35.227092    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 29 14:35:35.227483 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:35.227117    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-service-ca-bundle podName:1a5f8182-95a6-4f4b-b40b-ea003c7be4dd nodeName:}" failed. No retries permitted until 2026-07-29 14:35:37.227103117 +0000 UTC m=+39.275561034 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-service-ca-bundle") pod "router-default-746f69dff9-2bdj7" (UID: "1a5f8182-95a6-4f4b-b40b-ea003c7be4dd") : configmap references non-existent config key: service-ca.crt
Jul 29 14:35:35.227483 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:35.227155    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dbfdc61-952e-49ee-a2b2-a484d295b1e3-metrics-tls podName:4dbfdc61-952e-49ee-a2b2-a484d295b1e3 nodeName:}" failed. No retries permitted until 2026-07-29 14:35:37.227144564 +0000 UTC m=+39.275602484 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4dbfdc61-952e-49ee-a2b2-a484d295b1e3-metrics-tls") pod "dns-default-4m4xj" (UID: "4dbfdc61-952e-49ee-a2b2-a484d295b1e3") : secret "dns-default-metrics-tls" not found
Jul 29 14:35:35.227483 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:35.227172    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-metrics-certs podName:1a5f8182-95a6-4f4b-b40b-ea003c7be4dd nodeName:}" failed. No retries permitted until 2026-07-29 14:35:37.227162403 +0000 UTC m=+39.275620323 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-metrics-certs") pod "router-default-746f69dff9-2bdj7" (UID: "1a5f8182-95a6-4f4b-b40b-ea003c7be4dd") : secret "router-metrics-certs-default" not found
Jul 29 14:35:36.662908 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:36.662192    2563 generic.go:356] "Generic (PLEG): container finished" podID="e666f634-f60c-47fb-952c-7e18a781609b" containerID="c0066aa98eb32155a99d6b96fac736df3508885236c1fce8da4cfc75749dd4ea" exitCode=0
Jul 29 14:35:36.662908 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:36.662274    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nwj5f" event={"ID":"e666f634-f60c-47fb-952c-7e18a781609b","Type":"ContainerDied","Data":"c0066aa98eb32155a99d6b96fac736df3508885236c1fce8da4cfc75749dd4ea"}
Jul 29 14:35:36.768214 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:36.767808    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-qlc8b"]
Jul 29 14:35:36.800801 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:36.800774    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-qlc8b"]
Jul 29 14:35:36.800948 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:36.800897    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-qlc8b"
Jul 29 14:35:36.804703 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:36.803308    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-tzdkm\""
Jul 29 14:35:36.804703 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:36.803530    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 29 14:35:36.804703 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:36.803826    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 29 14:35:36.921732 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:36.921096    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2jpl_9a100b51-c157-4a63-a6c5-9d92a3fee8a7/konnectivity-agent/0.log" limit=1024
Jul 29 14:35:36.943924 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:36.943662    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6a44e5c-e648-4bb8-a393-7cfb4c22d761-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-qlc8b\" (UID: \"b6a44e5c-e648-4bb8-a393-7cfb4c22d761\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qlc8b"
Jul 29 14:35:36.943924 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:36.943718    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b6a44e5c-e648-4bb8-a393-7cfb4c22d761-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-qlc8b\" (UID: \"b6a44e5c-e648-4bb8-a393-7cfb4c22d761\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qlc8b"
Jul 29 14:35:36.999289 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:36.999229    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2jpl_9a100b51-c157-4a63-a6c5-9d92a3fee8a7/konnectivity-agent/0.log" limit=1024
Jul 29 14:35:37.046289 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:37.044738    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6a44e5c-e648-4bb8-a393-7cfb4c22d761-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-qlc8b\" (UID: \"b6a44e5c-e648-4bb8-a393-7cfb4c22d761\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qlc8b"
Jul 29 14:35:37.046289 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:37.044795    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b6a44e5c-e648-4bb8-a393-7cfb4c22d761-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-qlc8b\" (UID: \"b6a44e5c-e648-4bb8-a393-7cfb4c22d761\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qlc8b"
Jul 29 14:35:37.046289 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:37.045875    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b6a44e5c-e648-4bb8-a393-7cfb4c22d761-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-qlc8b\" (UID: \"b6a44e5c-e648-4bb8-a393-7cfb4c22d761\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qlc8b"
Jul 29 14:35:37.046289 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:37.045987    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 14:35:37.046289 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:37.046044    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6a44e5c-e648-4bb8-a393-7cfb4c22d761-networking-console-plugin-cert podName:b6a44e5c-e648-4bb8-a393-7cfb4c22d761 nodeName:}" failed. No retries permitted until 2026-07-29 14:35:37.54602678 +0000 UTC m=+39.594484699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b6a44e5c-e648-4bb8-a393-7cfb4c22d761-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-qlc8b" (UID: "b6a44e5c-e648-4bb8-a393-7cfb4c22d761") : secret "networking-console-plugin-cert" not found
Jul 29 14:35:37.060959 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:37.060902    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2jpl_9a100b51-c157-4a63-a6c5-9d92a3fee8a7/konnectivity-agent/0.log" limit=1024
Jul 29 14:35:37.147008 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:37.146061    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-registry-tls\") pod \"image-registry-5d8957c4d5-4pb5r\" (UID: \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\") " pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r"
Jul 29 14:35:37.147008 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:37.146448    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 14:35:37.147008 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:37.146465    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d8957c4d5-4pb5r: secret "image-registry-tls" not found
Jul 29 14:35:37.147008 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:37.146526    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-registry-tls podName:51a472c7-19c4-4c5c-9813-ebf6cb7dd70c nodeName:}" failed. No retries permitted until 2026-07-29 14:35:41.146506146 +0000 UTC m=+43.194964083 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-registry-tls") pod "image-registry-5d8957c4d5-4pb5r" (UID: "51a472c7-19c4-4c5c-9813-ebf6cb7dd70c") : secret "image-registry-tls" not found
Jul 29 14:35:37.248689 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:37.247620    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2037177-a695-47b3-bda0-0669bcfe408e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-bxjbs\" (UID: \"f2037177-a695-47b3-bda0-0669bcfe408e\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bxjbs"
Jul 29 14:35:37.248689 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:37.247672    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f010d9fc-7827-43c6-b9d8-03b9b09222ad-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-67fqk\" (UID: \"f010d9fc-7827-43c6-b9d8-03b9b09222ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-67fqk"
Jul 29 14:35:37.248689 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:37.247729    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4dbfdc61-952e-49ee-a2b2-a484d295b1e3-metrics-tls\") pod \"dns-default-4m4xj\" (UID: \"4dbfdc61-952e-49ee-a2b2-a484d295b1e3\") " pod="openshift-dns/dns-default-4m4xj"
Jul 29 14:35:37.248689 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:37.247817    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-metrics-certs\") pod \"router-default-746f69dff9-2bdj7\" (UID: \"1a5f8182-95a6-4f4b-b40b-ea003c7be4dd\") " pod="openshift-ingress/router-default-746f69dff9-2bdj7"
Jul 29 14:35:37.248689 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:37.247848    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-service-ca-bundle\") pod \"router-default-746f69dff9-2bdj7\" (UID: \"1a5f8182-95a6-4f4b-b40b-ea003c7be4dd\") " pod="openshift-ingress/router-default-746f69dff9-2bdj7"
Jul 29 14:35:37.248689 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:37.247997    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-service-ca-bundle podName:1a5f8182-95a6-4f4b-b40b-ea003c7be4dd nodeName:}" failed. No retries permitted until 2026-07-29 14:35:41.247979203 +0000 UTC m=+43.296437126 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-service-ca-bundle") pod "router-default-746f69dff9-2bdj7" (UID: "1a5f8182-95a6-4f4b-b40b-ea003c7be4dd") : configmap references non-existent config key: service-ca.crt
Jul 29 14:35:37.248689 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:37.248413    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 14:35:37.248689 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:37.248462    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2037177-a695-47b3-bda0-0669bcfe408e-cluster-monitoring-operator-tls podName:f2037177-a695-47b3-bda0-0669bcfe408e nodeName:}" failed. No retries permitted until 2026-07-29 14:35:41.248447445 +0000 UTC m=+43.296905379 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f2037177-a695-47b3-bda0-0669bcfe408e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-bxjbs" (UID: "f2037177-a695-47b3-bda0-0669bcfe408e") : secret "cluster-monitoring-operator-tls" not found
Jul 29 14:35:37.248689 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:37.248480    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 29 14:35:37.248689 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:37.248516    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-metrics-certs podName:1a5f8182-95a6-4f4b-b40b-ea003c7be4dd nodeName:}" failed. No retries permitted until 2026-07-29 14:35:41.248504779 +0000 UTC m=+43.296962701 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-metrics-certs") pod "router-default-746f69dff9-2bdj7" (UID: "1a5f8182-95a6-4f4b-b40b-ea003c7be4dd") : secret "router-metrics-certs-default" not found
Jul 29 14:35:37.248689 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:37.248573    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 14:35:37.248689 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:37.248606    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f010d9fc-7827-43c6-b9d8-03b9b09222ad-samples-operator-tls podName:f010d9fc-7827-43c6-b9d8-03b9b09222ad nodeName:}" failed. No retries permitted until 2026-07-29 14:35:41.248595584 +0000 UTC m=+43.297053505 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f010d9fc-7827-43c6-b9d8-03b9b09222ad-samples-operator-tls") pod "cluster-samples-operator-6748489656-67fqk" (UID: "f010d9fc-7827-43c6-b9d8-03b9b09222ad") : secret "samples-operator-tls" not found
Jul 29 14:35:37.248689 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:37.248629    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 14:35:37.248689 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:37.248662    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dbfdc61-952e-49ee-a2b2-a484d295b1e3-metrics-tls podName:4dbfdc61-952e-49ee-a2b2-a484d295b1e3 nodeName:}" failed. No retries permitted until 2026-07-29 14:35:41.248651095 +0000 UTC m=+43.297109014 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4dbfdc61-952e-49ee-a2b2-a484d295b1e3-metrics-tls") pod "dns-default-4m4xj" (UID: "4dbfdc61-952e-49ee-a2b2-a484d295b1e3") : secret "dns-default-metrics-tls" not found
Jul 29 14:35:37.551269 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:37.551184    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6a44e5c-e648-4bb8-a393-7cfb4c22d761-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-qlc8b\" (UID: \"b6a44e5c-e648-4bb8-a393-7cfb4c22d761\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qlc8b"
Jul 29 14:35:37.551437 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:37.551337    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 14:35:37.551437 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:37.551418    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6a44e5c-e648-4bb8-a393-7cfb4c22d761-networking-console-plugin-cert podName:b6a44e5c-e648-4bb8-a393-7cfb4c22d761 nodeName:}" failed. No retries permitted until 2026-07-29 14:35:38.551399149 +0000 UTC m=+40.599857070 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b6a44e5c-e648-4bb8-a393-7cfb4c22d761-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-qlc8b" (UID: "b6a44e5c-e648-4bb8-a393-7cfb4c22d761") : secret "networking-console-plugin-cert" not found
Jul 29 14:35:37.680726 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:37.680640    2563 generic.go:356] "Generic (PLEG): container finished" podID="e666f634-f60c-47fb-952c-7e18a781609b" containerID="1ecb5ebd719dbfc2c4b5a146fd37813acd4658c4b3b1f40e8ebf2f432d7a726a" exitCode=0
Jul 29 14:35:37.681179 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:37.680760    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nwj5f" event={"ID":"e666f634-f60c-47fb-952c-7e18a781609b","Type":"ContainerDied","Data":"1ecb5ebd719dbfc2c4b5a146fd37813acd4658c4b3b1f40e8ebf2f432d7a726a"}
Jul 29 14:35:38.561842 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:38.561797    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6a44e5c-e648-4bb8-a393-7cfb4c22d761-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-qlc8b\" (UID: \"b6a44e5c-e648-4bb8-a393-7cfb4c22d761\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qlc8b"
Jul 29 14:35:38.562017 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:38.561961    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 14:35:38.562099 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:38.562037    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6a44e5c-e648-4bb8-a393-7cfb4c22d761-networking-console-plugin-cert podName:b6a44e5c-e648-4bb8-a393-7cfb4c22d761 nodeName:}" failed. No retries permitted until 2026-07-29 14:35:40.562019834 +0000 UTC m=+42.610477752 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b6a44e5c-e648-4bb8-a393-7cfb4c22d761-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-qlc8b" (UID: "b6a44e5c-e648-4bb8-a393-7cfb4c22d761") : secret "networking-console-plugin-cert" not found
Jul 29 14:35:38.687707 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:38.687665    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nwj5f" event={"ID":"e666f634-f60c-47fb-952c-7e18a781609b","Type":"ContainerStarted","Data":"5e2d259dff83d2234be8d1326e99cc41ff905eadb8ed6984a6933e9313cf869f"}
Jul 29 14:35:38.710578 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:38.710520    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nwj5f" podStartSLOduration=7.672471873 podStartE2EDuration="40.710501745s" totalImagesPullingTime="33.038029872s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 14:34:58 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:35:38.709144474 +0000 UTC m=+40.757602405" watchObservedRunningTime="2026-07-29 14:35:38.710501745 +0000 UTC m=+40.758959684"
Jul 29 14:35:38.839091 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:38.839010    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2jpl_9a100b51-c157-4a63-a6c5-9d92a3fee8a7/konnectivity-agent/0.log" limit=1024
Jul 29 14:35:38.911089 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:38.911047    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2jpl_9a100b51-c157-4a63-a6c5-9d92a3fee8a7/konnectivity-agent/0.log" limit=1024
Jul 29 14:35:39.110838 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:39.110759    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2jpl_9a100b51-c157-4a63-a6c5-9d92a3fee8a7/konnectivity-agent/0.log" limit=1024
Jul 29 14:35:39.170717 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:39.170658    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2jpl_9a100b51-c157-4a63-a6c5-9d92a3fee8a7/konnectivity-agent/0.log" limit=1024
Jul 29 14:35:40.581084 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:40.581039    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6a44e5c-e648-4bb8-a393-7cfb4c22d761-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-qlc8b\" (UID: \"b6a44e5c-e648-4bb8-a393-7cfb4c22d761\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qlc8b"
Jul 29 14:35:40.581595 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:40.581214    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 14:35:40.581595 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:40.581280    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6a44e5c-e648-4bb8-a393-7cfb4c22d761-networking-console-plugin-cert podName:b6a44e5c-e648-4bb8-a393-7cfb4c22d761 nodeName:}" failed. No retries permitted until 2026-07-29 14:35:44.581264758 +0000 UTC m=+46.629722676 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b6a44e5c-e648-4bb8-a393-7cfb4c22d761-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-qlc8b" (UID: "b6a44e5c-e648-4bb8-a393-7cfb4c22d761") : secret "networking-console-plugin-cert" not found
Jul 29 14:35:41.186088 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:41.186055    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-registry-tls\") pod \"image-registry-5d8957c4d5-4pb5r\" (UID: \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\") " pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r"
Jul 29 14:35:41.186251 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:41.186222    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 14:35:41.186251 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:41.186244    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d8957c4d5-4pb5r: secret "image-registry-tls" not found
Jul 29 14:35:41.186342 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:41.186300    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-registry-tls podName:51a472c7-19c4-4c5c-9813-ebf6cb7dd70c nodeName:}" failed. No retries permitted until 2026-07-29 14:35:49.186284083 +0000 UTC m=+51.234742004 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-registry-tls") pod "image-registry-5d8957c4d5-4pb5r" (UID: "51a472c7-19c4-4c5c-9813-ebf6cb7dd70c") : secret "image-registry-tls" not found
Jul 29 14:35:41.287218 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:41.287181    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4dbfdc61-952e-49ee-a2b2-a484d295b1e3-metrics-tls\") pod \"dns-default-4m4xj\" (UID: \"4dbfdc61-952e-49ee-a2b2-a484d295b1e3\") " pod="openshift-dns/dns-default-4m4xj"
Jul 29 14:35:41.287366 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:41.287257    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-metrics-certs\") pod \"router-default-746f69dff9-2bdj7\" (UID: \"1a5f8182-95a6-4f4b-b40b-ea003c7be4dd\") " pod="openshift-ingress/router-default-746f69dff9-2bdj7"
Jul 29 14:35:41.287366 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:41.287275    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-service-ca-bundle\") pod \"router-default-746f69dff9-2bdj7\" (UID: \"1a5f8182-95a6-4f4b-b40b-ea003c7be4dd\") " pod="openshift-ingress/router-default-746f69dff9-2bdj7"
Jul 29 14:35:41.287366 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:41.287318    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2037177-a695-47b3-bda0-0669bcfe408e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-bxjbs\" (UID: \"f2037177-a695-47b3-bda0-0669bcfe408e\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bxjbs"
Jul 29 14:35:41.287366 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:41.287330    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 14:35:41.287590 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:41.287388    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 29 14:35:41.287590 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:41.287406    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dbfdc61-952e-49ee-a2b2-a484d295b1e3-metrics-tls podName:4dbfdc61-952e-49ee-a2b2-a484d295b1e3 nodeName:}" failed. No retries permitted until 2026-07-29 14:35:49.287390889 +0000 UTC m=+51.335848806 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4dbfdc61-952e-49ee-a2b2-a484d295b1e3-metrics-tls") pod "dns-default-4m4xj" (UID: "4dbfdc61-952e-49ee-a2b2-a484d295b1e3") : secret "dns-default-metrics-tls" not found
Jul 29 14:35:41.287590 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:41.287414    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 14:35:41.287590 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:41.287339    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f010d9fc-7827-43c6-b9d8-03b9b09222ad-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-67fqk\" (UID: \"f010d9fc-7827-43c6-b9d8-03b9b09222ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-67fqk"
Jul 29 14:35:41.287590 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:41.287436    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-service-ca-bundle podName:1a5f8182-95a6-4f4b-b40b-ea003c7be4dd nodeName:}" failed. No retries permitted until 2026-07-29 14:35:49.287413607 +0000 UTC m=+51.335871527 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-service-ca-bundle") pod "router-default-746f69dff9-2bdj7" (UID: "1a5f8182-95a6-4f4b-b40b-ea003c7be4dd") : configmap references non-existent config key: service-ca.crt
Jul 29 14:35:41.287590 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:41.287416    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 14:35:41.287590 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:41.287535    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-metrics-certs podName:1a5f8182-95a6-4f4b-b40b-ea003c7be4dd nodeName:}" failed. No retries permitted until 2026-07-29 14:35:49.287520612 +0000 UTC m=+51.335978535 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-metrics-certs") pod "router-default-746f69dff9-2bdj7" (UID: "1a5f8182-95a6-4f4b-b40b-ea003c7be4dd") : secret "router-metrics-certs-default" not found
Jul 29 14:35:41.287590 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:41.287551    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f010d9fc-7827-43c6-b9d8-03b9b09222ad-samples-operator-tls podName:f010d9fc-7827-43c6-b9d8-03b9b09222ad nodeName:}" failed. No retries permitted until 2026-07-29 14:35:49.287542625 +0000 UTC m=+51.336000549 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f010d9fc-7827-43c6-b9d8-03b9b09222ad-samples-operator-tls") pod "cluster-samples-operator-6748489656-67fqk" (UID: "f010d9fc-7827-43c6-b9d8-03b9b09222ad") : secret "samples-operator-tls" not found
Jul 29 14:35:41.287590 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:41.287567    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2037177-a695-47b3-bda0-0669bcfe408e-cluster-monitoring-operator-tls podName:f2037177-a695-47b3-bda0-0669bcfe408e nodeName:}" failed. No retries permitted until 2026-07-29 14:35:49.287558165 +0000 UTC m=+51.336016085 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f2037177-a695-47b3-bda0-0669bcfe408e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-bxjbs" (UID: "f2037177-a695-47b3-bda0-0669bcfe408e") : secret "cluster-monitoring-operator-tls" not found
Jul 29 14:35:43.706618 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:43.706592    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qcjfr" event={"ID":"031b9fd3-1851-451a-b5e4-7f667c641263","Type":"ContainerStarted","Data":"bad5093b2557a67b80dacb755100f07ae6fc70d2f09fcedf5b30b38603e0f826"}
Jul 29 14:35:43.708287 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:43.708268    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-g9dbl" event={"ID":"c1eb675a-8bbd-48ef-a526-506a7fd20ee7","Type":"ContainerStarted","Data":"e0ec0e1e30d32351b3da73ff34f5df854f4c43162513a125e56714ccc227fc11"}
Jul 29 14:35:43.738913 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:43.738866    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qcjfr" podStartSLOduration=1.264528165 podStartE2EDuration="10.738849515s" totalImagesPullingTime="9.47432135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:35:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:35:43.738328878 +0000 UTC m=+45.786786819" watchObservedRunningTime="2026-07-29 14:35:43.738849515 +0000 UTC m=+45.787307454"
Jul 29 14:35:43.753061 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:43.752902    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-g9dbl" podStartSLOduration=23.599123906 podStartE2EDuration="32.752881058s" totalImagesPullingTime="9.153757152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:35:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:35:43.751483204 +0000 UTC m=+45.799941144" watchObservedRunningTime="2026-07-29 14:35:43.752881058 +0000 UTC m=+45.801338998"
Jul 29 14:35:43.829080 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:43.829028    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2jpl_9a100b51-c157-4a63-a6c5-9d92a3fee8a7/konnectivity-agent/0.log" limit=1024
Jul 29 14:35:43.931550 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:43.931460    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2jpl_9a100b51-c157-4a63-a6c5-9d92a3fee8a7/konnectivity-agent/0.log" limit=1024
Jul 29 14:35:44.625402 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:44.625366    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6a44e5c-e648-4bb8-a393-7cfb4c22d761-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-qlc8b\" (UID: \"b6a44e5c-e648-4bb8-a393-7cfb4c22d761\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qlc8b"
Jul 29 14:35:44.625917 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:44.625744    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 14:35:44.625917 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:44.625816    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6a44e5c-e648-4bb8-a393-7cfb4c22d761-networking-console-plugin-cert podName:b6a44e5c-e648-4bb8-a393-7cfb4c22d761 nodeName:}" failed. No retries permitted until 2026-07-29 14:35:52.625796006 +0000 UTC m=+54.674253930 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b6a44e5c-e648-4bb8-a393-7cfb4c22d761-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-qlc8b" (UID: "b6a44e5c-e648-4bb8-a393-7cfb4c22d761") : secret "networking-console-plugin-cert" not found
Jul 29 14:35:44.634776 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:44.634748    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mvd8l"]
Jul 29 14:35:44.655347 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:44.655312    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mvd8l"]
Jul 29 14:35:44.655480 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:44.655440    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mvd8l"
Jul 29 14:35:44.657384 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:44.657327    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 29 14:35:44.657577 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:44.657383    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 29 14:35:44.657577 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:44.657412    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zhf56\""
Jul 29 14:35:44.672627 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:44.672597    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2jpl_9a100b51-c157-4a63-a6c5-9d92a3fee8a7/konnectivity-agent/0.log" limit=1024
Jul 29 14:35:44.714643 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:44.714606    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-lbmwl" event={"ID":"5f530514-b6e9-4e5e-9f31-4d6a0792f742","Type":"ContainerStarted","Data":"e45594a893f4194cd2d123575943785bb493fcf8cb6183c62829179f314feba7"}
Jul 29 14:35:44.716421 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:44.716391    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-nh8pd" event={"ID":"15611a65-9454-4396-9b2f-fc9599672fc4","Type":"ContainerStarted","Data":"b9e4bb84e6bf82d686e336e7ce65cbf1fd2b036cd0e44f032b5c14161aec98dc"}
Jul 29 14:35:44.718808 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:44.718772    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9j4jg" event={"ID":"cca0fc93-ab3f-4cb2-aaab-8fcdc4428636","Type":"ContainerStarted","Data":"465d70de46cffa037ec1869b7184acd88cdb73aea39404ec32cc60a93b09867e"}
Jul 29 14:35:44.720823 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:44.720796    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-fkjfz" event={"ID":"7504b97a-38e2-44ff-822a-a7f1e98b8933","Type":"ContainerStarted","Data":"417e1d881d2eb45cb8401f8d06b7a2a521a365994b4cf1120efec88221eb8243"}
Jul 29 14:35:44.726358 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:44.726325    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d86b4f92-11dc-402c-80cd-53bbb1981778-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mvd8l\" (UID: \"d86b4f92-11dc-402c-80cd-53bbb1981778\") " pod="openshift-insights/insights-runtime-extractor-mvd8l"
Jul 29 14:35:44.726449 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:44.726398    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk6vt\" (UniqueName: \"kubernetes.io/projected/d86b4f92-11dc-402c-80cd-53bbb1981778-kube-api-access-mk6vt\") pod \"insights-runtime-extractor-mvd8l\" (UID: \"d86b4f92-11dc-402c-80cd-53bbb1981778\") " pod="openshift-insights/insights-runtime-extractor-mvd8l"
Jul 29 14:35:44.726498 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:44.726446    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d86b4f92-11dc-402c-80cd-53bbb1981778-crio-socket\") pod \"insights-runtime-extractor-mvd8l\" (UID: \"d86b4f92-11dc-402c-80cd-53bbb1981778\") " pod="openshift-insights/insights-runtime-extractor-mvd8l"
Jul 29 14:35:44.726498 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:44.726471    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d86b4f92-11dc-402c-80cd-53bbb1981778-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mvd8l\" (UID: \"d86b4f92-11dc-402c-80cd-53bbb1981778\") " pod="openshift-insights/insights-runtime-extractor-mvd8l"
Jul 29 14:35:44.726559 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:44.726537    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d86b4f92-11dc-402c-80cd-53bbb1981778-data-volume\") pod \"insights-runtime-extractor-mvd8l\" (UID: \"d86b4f92-11dc-402c-80cd-53bbb1981778\") " pod="openshift-insights/insights-runtime-extractor-mvd8l"
Jul 29 14:35:44.727225 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:44.727202    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2jpl_9a100b51-c157-4a63-a6c5-9d92a3fee8a7/konnectivity-agent/0.log" limit=1024
Jul 29 14:35:44.727643 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:44.727618    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xnm95_23d79588-937d-4c6d-8760-a081024f608e/console-operator/0.log"
Jul 29 14:35:44.727710 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:44.727656    2563 generic.go:356] "Generic (PLEG): container finished" podID="23d79588-937d-4c6d-8760-a081024f608e" containerID="2193bef366b33695c42e603c573c846c508d1ecd1b133bde7188c50bc18af0ff" exitCode=255
Jul 29 14:35:44.727894 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:44.727856    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-xnm95" event={"ID":"23d79588-937d-4c6d-8760-a081024f608e","Type":"ContainerDied","Data":"2193bef366b33695c42e603c573c846c508d1ecd1b133bde7188c50bc18af0ff"}
Jul 29 14:35:44.728155 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:44.728125    2563 scope.go:117] "RemoveContainer" containerID="2193bef366b33695c42e603c573c846c508d1ecd1b133bde7188c50bc18af0ff"
Jul 29 14:35:44.733368 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:44.733325    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-lbmwl" podStartSLOduration=24.227694147 podStartE2EDuration="33.733307726s" totalImagesPullingTime="9.505613579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:35:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:35:44.733065487 +0000 UTC m=+46.781523427" watchObservedRunningTime="2026-07-29 14:35:44.733307726 +0000 UTC m=+46.781765666"
Jul 29 14:35:44.751483 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:44.751423    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-nh8pd" podStartSLOduration=24.194440605 podStartE2EDuration="33.751400439s" totalImagesPullingTime="9.556959834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:35:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:35:44.751123338 +0000 UTC m=+46.799581277" watchObservedRunningTime="2026-07-29 14:35:44.751400439 +0000 UTC m=+46.799858383"
Jul 29 14:35:44.783989 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:44.783926    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9j4jg" podStartSLOduration=24.308271993 podStartE2EDuration="33.783905572s" totalImagesPullingTime="9.475633579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:35:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:35:44.782766898 +0000 UTC m=+46.831224838" watchObservedRunningTime="2026-07-29 14:35:44.783905572 +0000 UTC m=+46.832363512"
Jul 29 14:35:44.796551 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:44.796498    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-fkjfz" podStartSLOduration=24.236525607 podStartE2EDuration="33.796479106s" totalImagesPullingTime="9.559953499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:35:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:35:44.795647589 +0000 UTC m=+46.844105531" watchObservedRunningTime="2026-07-29 14:35:44.796479106 +0000 UTC m=+46.844937046"
Jul 29 14:35:44.827300 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:44.827201    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d86b4f92-11dc-402c-80cd-53bbb1981778-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mvd8l\" (UID: \"d86b4f92-11dc-402c-80cd-53bbb1981778\") " pod="openshift-insights/insights-runtime-extractor-mvd8l"
Jul 29 14:35:44.827300 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:44.827263    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk6vt\" (UniqueName: \"kubernetes.io/projected/d86b4f92-11dc-402c-80cd-53bbb1981778-kube-api-access-mk6vt\") pod \"insights-runtime-extractor-mvd8l\" (UID: \"d86b4f92-11dc-402c-80cd-53bbb1981778\") " pod="openshift-insights/insights-runtime-extractor-mvd8l"
Jul 29 14:35:44.827517 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:44.827367    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d86b4f92-11dc-402c-80cd-53bbb1981778-crio-socket\") pod \"insights-runtime-extractor-mvd8l\" (UID: \"d86b4f92-11dc-402c-80cd-53bbb1981778\") " pod="openshift-insights/insights-runtime-extractor-mvd8l"
Jul 29 14:35:44.827517 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:44.827392    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d86b4f92-11dc-402c-80cd-53bbb1981778-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mvd8l\" (UID: \"d86b4f92-11dc-402c-80cd-53bbb1981778\") " pod="openshift-insights/insights-runtime-extractor-mvd8l"
Jul 29 14:35:44.827517 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:44.827502    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d86b4f92-11dc-402c-80cd-53bbb1981778-data-volume\") pod \"insights-runtime-extractor-mvd8l\" (UID: \"d86b4f92-11dc-402c-80cd-53bbb1981778\") " pod="openshift-insights/insights-runtime-extractor-mvd8l"
Jul 29 14:35:44.830424 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:44.828933    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d86b4f92-11dc-402c-80cd-53bbb1981778-data-volume\") pod \"insights-runtime-extractor-mvd8l\" (UID: \"d86b4f92-11dc-402c-80cd-53bbb1981778\") " pod="openshift-insights/insights-runtime-extractor-mvd8l"
Jul 29 14:35:44.830424 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:44.829614    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 14:35:44.830424 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:44.829687    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d86b4f92-11dc-402c-80cd-53bbb1981778-kube-rbac-proxy-cm podName:d86b4f92-11dc-402c-80cd-53bbb1981778 nodeName:}" failed. No retries permitted until 2026-07-29 14:35:45.329665443 +0000 UTC m=+47.378123363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d86b4f92-11dc-402c-80cd-53bbb1981778-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mvd8l" (UID: "d86b4f92-11dc-402c-80cd-53bbb1981778") : configmap "kube-rbac-proxy" not found
Jul 29 14:35:44.830424 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:44.829952    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 14:35:44.830424 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:44.830004    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d86b4f92-11dc-402c-80cd-53bbb1981778-insights-runtime-extractor-tls podName:d86b4f92-11dc-402c-80cd-53bbb1981778 nodeName:}" failed. No retries permitted until 2026-07-29 14:35:45.329986438 +0000 UTC m=+47.378444358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d86b4f92-11dc-402c-80cd-53bbb1981778-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mvd8l" (UID: "d86b4f92-11dc-402c-80cd-53bbb1981778") : secret "insights-runtime-extractor-tls" not found
Jul 29 14:35:44.830424 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:44.830005    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d86b4f92-11dc-402c-80cd-53bbb1981778-crio-socket\") pod \"insights-runtime-extractor-mvd8l\" (UID: \"d86b4f92-11dc-402c-80cd-53bbb1981778\") " pod="openshift-insights/insights-runtime-extractor-mvd8l"
Jul 29 14:35:44.841583 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:44.841556    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk6vt\" (UniqueName: \"kubernetes.io/projected/d86b4f92-11dc-402c-80cd-53bbb1981778-kube-api-access-mk6vt\") pod \"insights-runtime-extractor-mvd8l\" (UID: \"d86b4f92-11dc-402c-80cd-53bbb1981778\") " pod="openshift-insights/insights-runtime-extractor-mvd8l"
Jul 29 14:35:45.333293 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:45.333253    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d86b4f92-11dc-402c-80cd-53bbb1981778-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mvd8l\" (UID: \"d86b4f92-11dc-402c-80cd-53bbb1981778\") " pod="openshift-insights/insights-runtime-extractor-mvd8l"
Jul 29 14:35:45.333466 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:45.333326    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d86b4f92-11dc-402c-80cd-53bbb1981778-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mvd8l\" (UID: \"d86b4f92-11dc-402c-80cd-53bbb1981778\") " pod="openshift-insights/insights-runtime-extractor-mvd8l"
Jul 29 14:35:45.333466 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:45.333386    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 14:35:45.333466 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:45.333463    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d86b4f92-11dc-402c-80cd-53bbb1981778-kube-rbac-proxy-cm podName:d86b4f92-11dc-402c-80cd-53bbb1981778 nodeName:}" failed. No retries permitted until 2026-07-29 14:35:46.333441837 +0000 UTC m=+48.381899759 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d86b4f92-11dc-402c-80cd-53bbb1981778-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mvd8l" (UID: "d86b4f92-11dc-402c-80cd-53bbb1981778") : configmap "kube-rbac-proxy" not found
Jul 29 14:35:45.333656 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:45.333464    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 14:35:45.333656 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:45.333516    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d86b4f92-11dc-402c-80cd-53bbb1981778-insights-runtime-extractor-tls podName:d86b4f92-11dc-402c-80cd-53bbb1981778 nodeName:}" failed. No retries permitted until 2026-07-29 14:35:46.333504184 +0000 UTC m=+48.381962118 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d86b4f92-11dc-402c-80cd-53bbb1981778-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mvd8l" (UID: "d86b4f92-11dc-402c-80cd-53bbb1981778") : secret "insights-runtime-extractor-tls" not found
Jul 29 14:35:45.438433 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:45.438397    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2jpl_9a100b51-c157-4a63-a6c5-9d92a3fee8a7/konnectivity-agent/0.log" limit=1024
Jul 29 14:35:45.545236 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:45.545199    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2jpl_9a100b51-c157-4a63-a6c5-9d92a3fee8a7/konnectivity-agent/0.log" limit=1024
Jul 29 14:35:45.611598 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:45.611524    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-hqn29"]
Jul 29 14:35:45.628992 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:45.628955    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-hqn29"]
Jul 29 14:35:45.629182 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:45.629000    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-hqn29"
Jul 29 14:35:45.631117 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:45.631073    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 29 14:35:45.631287 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:45.631263    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6gwc7\""
Jul 29 14:35:45.631829 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:45.631776    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 29 14:35:45.648094 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:45.648062    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2jpl_9a100b51-c157-4a63-a6c5-9d92a3fee8a7/konnectivity-agent/0.log" limit=1024
Jul 29 14:35:45.732562 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:45.732535    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xnm95_23d79588-937d-4c6d-8760-a081024f608e/console-operator/1.log"
Jul 29 14:35:45.733069 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:45.733052    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xnm95_23d79588-937d-4c6d-8760-a081024f608e/console-operator/0.log"
Jul 29 14:35:45.733107 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:45.733086    2563 generic.go:356] "Generic (PLEG): container finished" podID="23d79588-937d-4c6d-8760-a081024f608e" containerID="cf45f8fc57d0be003cd9ad212f6b72af584a8fedc2f87a05a7cd6af981bfbe4c" exitCode=255
Jul 29 14:35:45.733206 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:45.733180    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-xnm95" event={"ID":"23d79588-937d-4c6d-8760-a081024f608e","Type":"ContainerDied","Data":"cf45f8fc57d0be003cd9ad212f6b72af584a8fedc2f87a05a7cd6af981bfbe4c"}
Jul 29 14:35:45.733259 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:45.733238    2563 scope.go:117] "RemoveContainer" containerID="2193bef366b33695c42e603c573c846c508d1ecd1b133bde7188c50bc18af0ff"
Jul 29 14:35:45.733467 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:45.733451    2563 scope.go:117] "RemoveContainer" containerID="cf45f8fc57d0be003cd9ad212f6b72af584a8fedc2f87a05a7cd6af981bfbe4c"
Jul 29 14:35:45.733701 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:45.733673    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-xnm95_openshift-console-operator(23d79588-937d-4c6d-8760-a081024f608e)\"" pod="openshift-console-operator/console-operator-584dcb6b54-xnm95" podUID="23d79588-937d-4c6d-8760-a081024f608e"
Jul 29 14:35:45.735739 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:45.735719    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm2kc\" (UniqueName: \"kubernetes.io/projected/14d576fd-12e6-4370-9499-2112ba0c9eb7-kube-api-access-tm2kc\") pod \"migrator-6865c9967-hqn29\" (UID: \"14d576fd-12e6-4370-9499-2112ba0c9eb7\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-hqn29"
Jul 29 14:35:45.739856 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:45.739822    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k2jpl_9a100b51-c157-4a63-a6c5-9d92a3fee8a7/konnectivity-agent/0.log" limit=1024
Jul 29 14:35:45.837316 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:45.837283    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm2kc\" (UniqueName: \"kubernetes.io/projected/14d576fd-12e6-4370-9499-2112ba0c9eb7-kube-api-access-tm2kc\") pod \"migrator-6865c9967-hqn29\" (UID: \"14d576fd-12e6-4370-9499-2112ba0c9eb7\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-hqn29"
Jul 29 14:35:45.848606 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:45.848576    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm2kc\" (UniqueName: \"kubernetes.io/projected/14d576fd-12e6-4370-9499-2112ba0c9eb7-kube-api-access-tm2kc\") pod \"migrator-6865c9967-hqn29\" (UID: \"14d576fd-12e6-4370-9499-2112ba0c9eb7\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-hqn29"
Jul 29 14:35:45.937725 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:45.937639    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-hqn29"
Jul 29 14:35:46.096268 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:46.096239    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-hqn29"]
Jul 29 14:35:46.099021 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:35:46.098992    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d576fd_12e6_4370_9499_2112ba0c9eb7.slice/crio-021fb4dd5be85fb1c06864ad81ee31ee0b650b7bc7e92ffe898c5146e463a1df WatchSource:0}: Error finding container 021fb4dd5be85fb1c06864ad81ee31ee0b650b7bc7e92ffe898c5146e463a1df: Status 404 returned error can't find the container with id 021fb4dd5be85fb1c06864ad81ee31ee0b650b7bc7e92ffe898c5146e463a1df
Jul 29 14:35:46.127589 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:46.127563    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n7qwt_8a010eab-ffa4-4f5a-a535-ecae3e55cf16/dns-node-resolver/0.log"
Jul 29 14:35:46.342754 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:46.342712    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d86b4f92-11dc-402c-80cd-53bbb1981778-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mvd8l\" (UID: \"d86b4f92-11dc-402c-80cd-53bbb1981778\") " pod="openshift-insights/insights-runtime-extractor-mvd8l"
Jul 29 14:35:46.342914 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:46.342777    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d86b4f92-11dc-402c-80cd-53bbb1981778-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mvd8l\" (UID: \"d86b4f92-11dc-402c-80cd-53bbb1981778\") " pod="openshift-insights/insights-runtime-extractor-mvd8l"
Jul 29 14:35:46.342914 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:46.342881    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 14:35:46.342914 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:46.342900    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 14:35:46.343043 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:46.342947    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d86b4f92-11dc-402c-80cd-53bbb1981778-insights-runtime-extractor-tls podName:d86b4f92-11dc-402c-80cd-53bbb1981778 nodeName:}" failed. No retries permitted until 2026-07-29 14:35:48.342930001 +0000 UTC m=+50.391387918 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d86b4f92-11dc-402c-80cd-53bbb1981778-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mvd8l" (UID: "d86b4f92-11dc-402c-80cd-53bbb1981778") : secret "insights-runtime-extractor-tls" not found
Jul 29 14:35:46.343043 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:46.342962    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d86b4f92-11dc-402c-80cd-53bbb1981778-kube-rbac-proxy-cm podName:d86b4f92-11dc-402c-80cd-53bbb1981778 nodeName:}" failed. No retries permitted until 2026-07-29 14:35:48.342955361 +0000 UTC m=+50.391413279 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d86b4f92-11dc-402c-80cd-53bbb1981778-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mvd8l" (UID: "d86b4f92-11dc-402c-80cd-53bbb1981778") : configmap "kube-rbac-proxy" not found
Jul 29 14:35:46.740620 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:46.740578    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-hqn29" event={"ID":"14d576fd-12e6-4370-9499-2112ba0c9eb7","Type":"ContainerStarted","Data":"021fb4dd5be85fb1c06864ad81ee31ee0b650b7bc7e92ffe898c5146e463a1df"}
Jul 29 14:35:46.742053 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:46.742032    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xnm95_23d79588-937d-4c6d-8760-a081024f608e/console-operator/1.log"
Jul 29 14:35:46.742398 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:46.742380    2563 scope.go:117] "RemoveContainer" containerID="cf45f8fc57d0be003cd9ad212f6b72af584a8fedc2f87a05a7cd6af981bfbe4c"
Jul 29 14:35:46.742558 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:46.742540    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-xnm95_openshift-console-operator(23d79588-937d-4c6d-8760-a081024f608e)\"" pod="openshift-console-operator/console-operator-584dcb6b54-xnm95" podUID="23d79588-937d-4c6d-8760-a081024f608e"
Jul 29 14:35:46.923929 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:46.923897    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d44hd_75a67018-8010-4684-ac1c-4a12c7f30bea/node-ca/0.log"
Jul 29 14:35:47.523996 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:47.523964    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qcjfr_031b9fd3-1851-451a-b5e4-7f667c641263/serve-healthcheck-canary/0.log"
Jul 29 14:35:47.843890 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:47.843820    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-5qlf4"]
Jul 29 14:35:47.858019 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:47.857992    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-5qlf4"]
Jul 29 14:35:47.858160 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:47.858095    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-5qlf4"
Jul 29 14:35:47.860513 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:47.860492    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 29 14:35:47.861020 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:47.860997    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 29 14:35:47.861147 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:47.861037    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-j5dhf\""
Jul 29 14:35:47.861147 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:47.861044    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 29 14:35:47.861147 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:47.861114    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 29 14:35:47.958693 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:47.958659    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/06472e5e-da44-41e6-90cb-cf21ff6a0793-signing-cabundle\") pod \"service-ca-658d7b7c78-5qlf4\" (UID: \"06472e5e-da44-41e6-90cb-cf21ff6a0793\") " pod="openshift-service-ca/service-ca-658d7b7c78-5qlf4"
Jul 29 14:35:47.958693 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:47.958698    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/06472e5e-da44-41e6-90cb-cf21ff6a0793-signing-key\") pod \"service-ca-658d7b7c78-5qlf4\" (UID: \"06472e5e-da44-41e6-90cb-cf21ff6a0793\") " pod="openshift-service-ca/service-ca-658d7b7c78-5qlf4"
Jul 29 14:35:47.958891 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:47.958725    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9qbg\" (UniqueName: \"kubernetes.io/projected/06472e5e-da44-41e6-90cb-cf21ff6a0793-kube-api-access-k9qbg\") pod \"service-ca-658d7b7c78-5qlf4\" (UID: \"06472e5e-da44-41e6-90cb-cf21ff6a0793\") " pod="openshift-service-ca/service-ca-658d7b7c78-5qlf4"
Jul 29 14:35:48.060063 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:48.060037    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/06472e5e-da44-41e6-90cb-cf21ff6a0793-signing-cabundle\") pod \"service-ca-658d7b7c78-5qlf4\" (UID: \"06472e5e-da44-41e6-90cb-cf21ff6a0793\") " pod="openshift-service-ca/service-ca-658d7b7c78-5qlf4"
Jul 29 14:35:48.060195 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:48.060072    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/06472e5e-da44-41e6-90cb-cf21ff6a0793-signing-key\") pod \"service-ca-658d7b7c78-5qlf4\" (UID: \"06472e5e-da44-41e6-90cb-cf21ff6a0793\") " pod="openshift-service-ca/service-ca-658d7b7c78-5qlf4"
Jul 29 14:35:48.060195 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:48.060097    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9qbg\" (UniqueName: \"kubernetes.io/projected/06472e5e-da44-41e6-90cb-cf21ff6a0793-kube-api-access-k9qbg\") pod \"service-ca-658d7b7c78-5qlf4\" (UID: \"06472e5e-da44-41e6-90cb-cf21ff6a0793\") " pod="openshift-service-ca/service-ca-658d7b7c78-5qlf4"
Jul 29 14:35:48.060873 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:48.060845    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/06472e5e-da44-41e6-90cb-cf21ff6a0793-signing-cabundle\") pod \"service-ca-658d7b7c78-5qlf4\" (UID: \"06472e5e-da44-41e6-90cb-cf21ff6a0793\") " pod="openshift-service-ca/service-ca-658d7b7c78-5qlf4"
Jul 29 14:35:48.062517 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:48.062497    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/06472e5e-da44-41e6-90cb-cf21ff6a0793-signing-key\") pod \"service-ca-658d7b7c78-5qlf4\" (UID: \"06472e5e-da44-41e6-90cb-cf21ff6a0793\") " pod="openshift-service-ca/service-ca-658d7b7c78-5qlf4"
Jul 29 14:35:48.070849 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:48.070826    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9qbg\" (UniqueName: \"kubernetes.io/projected/06472e5e-da44-41e6-90cb-cf21ff6a0793-kube-api-access-k9qbg\") pod \"service-ca-658d7b7c78-5qlf4\" (UID: \"06472e5e-da44-41e6-90cb-cf21ff6a0793\") " pod="openshift-service-ca/service-ca-658d7b7c78-5qlf4"
Jul 29 14:35:48.166975 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:48.166944    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-5qlf4"
Jul 29 14:35:48.309777 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:48.309747    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-5qlf4"]
Jul 29 14:35:48.312684 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:35:48.312658    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06472e5e_da44_41e6_90cb_cf21ff6a0793.slice/crio-f7fc1fa207460f1a84ab37261c9d34e27a76ce82faf1b44e6c7679754cc03cab WatchSource:0}: Error finding container f7fc1fa207460f1a84ab37261c9d34e27a76ce82faf1b44e6c7679754cc03cab: Status 404 returned error can't find the container with id f7fc1fa207460f1a84ab37261c9d34e27a76ce82faf1b44e6c7679754cc03cab
Jul 29 14:35:48.362359 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:48.362333    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d86b4f92-11dc-402c-80cd-53bbb1981778-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mvd8l\" (UID: \"d86b4f92-11dc-402c-80cd-53bbb1981778\") " pod="openshift-insights/insights-runtime-extractor-mvd8l"
Jul 29 14:35:48.362500 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:48.362481    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 14:35:48.362573 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:48.362536    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d86b4f92-11dc-402c-80cd-53bbb1981778-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mvd8l\" (UID: \"d86b4f92-11dc-402c-80cd-53bbb1981778\") " pod="openshift-insights/insights-runtime-extractor-mvd8l"
Jul 29 14:35:48.362626 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:48.362546    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d86b4f92-11dc-402c-80cd-53bbb1981778-insights-runtime-extractor-tls podName:d86b4f92-11dc-402c-80cd-53bbb1981778 nodeName:}" failed. No retries permitted until 2026-07-29 14:35:52.362530112 +0000 UTC m=+54.410988032 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d86b4f92-11dc-402c-80cd-53bbb1981778-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mvd8l" (UID: "d86b4f92-11dc-402c-80cd-53bbb1981778") : secret "insights-runtime-extractor-tls" not found
Jul 29 14:35:48.362687 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:48.362642    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 14:35:48.362738 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:48.362708    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d86b4f92-11dc-402c-80cd-53bbb1981778-kube-rbac-proxy-cm podName:d86b4f92-11dc-402c-80cd-53bbb1981778 nodeName:}" failed. No retries permitted until 2026-07-29 14:35:52.36269093 +0000 UTC m=+54.411148851 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d86b4f92-11dc-402c-80cd-53bbb1981778-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mvd8l" (UID: "d86b4f92-11dc-402c-80cd-53bbb1981778") : configmap "kube-rbac-proxy" not found
Jul 29 14:35:48.750696 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:48.750662    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-hqn29" event={"ID":"14d576fd-12e6-4370-9499-2112ba0c9eb7","Type":"ContainerStarted","Data":"efe3658ed29876882d589473fbe77f949300bdd5586cc30d5a5bb1905ae01e4d"}
Jul 29 14:35:48.750883 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:48.750711    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-hqn29" event={"ID":"14d576fd-12e6-4370-9499-2112ba0c9eb7","Type":"ContainerStarted","Data":"2713fb1c41814a705f63de0ed198a8523fe9f94858df1a92f9a6afb344ee644d"}
Jul 29 14:35:48.752125 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:48.752100    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-5qlf4" event={"ID":"06472e5e-da44-41e6-90cb-cf21ff6a0793","Type":"ContainerStarted","Data":"2fd25ad3055deffafeaab2dba124a7cf2c657b4af6da6f3be8e23ba4287d41ed"}
Jul 29 14:35:48.752251 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:48.752152    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-5qlf4" event={"ID":"06472e5e-da44-41e6-90cb-cf21ff6a0793","Type":"ContainerStarted","Data":"f7fc1fa207460f1a84ab37261c9d34e27a76ce82faf1b44e6c7679754cc03cab"}
Jul 29 14:35:48.774631 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:48.774585    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-hqn29" podStartSLOduration=1.873949631 podStartE2EDuration="3.774569841s" totalImagesPullingTime="1.90062021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:35:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:35:48.774402635 +0000 UTC m=+50.822860575" watchObservedRunningTime="2026-07-29 14:35:48.774569841 +0000 UTC m=+50.823027838"
Jul 29 14:35:48.793200 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:48.793149    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-5qlf4" podStartSLOduration=1.7931137339999998 podStartE2EDuration="1.793113734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:35:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:35:48.792200765 +0000 UTC m=+50.840658707" watchObservedRunningTime="2026-07-29 14:35:48.793113734 +0000 UTC m=+50.841571672"
Jul 29 14:35:49.270298 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:49.270260    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-registry-tls\") pod \"image-registry-5d8957c4d5-4pb5r\" (UID: \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\") " pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r"
Jul 29 14:35:49.270772 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:49.270431    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 14:35:49.270772 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:49.270452    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d8957c4d5-4pb5r: secret "image-registry-tls" not found
Jul 29 14:35:49.270772 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:49.270509    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-registry-tls podName:51a472c7-19c4-4c5c-9813-ebf6cb7dd70c nodeName:}" failed. No retries permitted until 2026-07-29 14:36:05.270494053 +0000 UTC m=+67.318951971 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-registry-tls") pod "image-registry-5d8957c4d5-4pb5r" (UID: "51a472c7-19c4-4c5c-9813-ebf6cb7dd70c") : secret "image-registry-tls" not found
Jul 29 14:35:49.371580 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:49.371539    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2037177-a695-47b3-bda0-0669bcfe408e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-bxjbs\" (UID: \"f2037177-a695-47b3-bda0-0669bcfe408e\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bxjbs"
Jul 29 14:35:49.371580 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:49.371583    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f010d9fc-7827-43c6-b9d8-03b9b09222ad-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-67fqk\" (UID: \"f010d9fc-7827-43c6-b9d8-03b9b09222ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-67fqk"
Jul 29 14:35:49.371823 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:49.371631    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4dbfdc61-952e-49ee-a2b2-a484d295b1e3-metrics-tls\") pod \"dns-default-4m4xj\" (UID: \"4dbfdc61-952e-49ee-a2b2-a484d295b1e3\") " pod="openshift-dns/dns-default-4m4xj"
Jul 29 14:35:49.371823 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:49.371709    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-metrics-certs\") pod \"router-default-746f69dff9-2bdj7\" (UID: \"1a5f8182-95a6-4f4b-b40b-ea003c7be4dd\") " pod="openshift-ingress/router-default-746f69dff9-2bdj7"
Jul 29 14:35:49.371823 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:49.371736    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-service-ca-bundle\") pod \"router-default-746f69dff9-2bdj7\" (UID: \"1a5f8182-95a6-4f4b-b40b-ea003c7be4dd\") " pod="openshift-ingress/router-default-746f69dff9-2bdj7"
Jul 29 14:35:49.371823 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:49.371708    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 14:35:49.371823 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:49.371789    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 14:35:49.371823 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:49.371808    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 29 14:35:49.372104 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:49.371851    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dbfdc61-952e-49ee-a2b2-a484d295b1e3-metrics-tls podName:4dbfdc61-952e-49ee-a2b2-a484d295b1e3 nodeName:}" failed. No retries permitted until 2026-07-29 14:36:05.371830287 +0000 UTC m=+67.420288319 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4dbfdc61-952e-49ee-a2b2-a484d295b1e3-metrics-tls") pod "dns-default-4m4xj" (UID: "4dbfdc61-952e-49ee-a2b2-a484d295b1e3") : secret "dns-default-metrics-tls" not found
Jul 29 14:35:49.372104 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:49.371811    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 14:35:49.372104 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:49.371869    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2037177-a695-47b3-bda0-0669bcfe408e-cluster-monitoring-operator-tls podName:f2037177-a695-47b3-bda0-0669bcfe408e nodeName:}" failed. No retries permitted until 2026-07-29 14:36:05.371860539 +0000 UTC m=+67.420318461 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f2037177-a695-47b3-bda0-0669bcfe408e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-bxjbs" (UID: "f2037177-a695-47b3-bda0-0669bcfe408e") : secret "cluster-monitoring-operator-tls" not found
Jul 29 14:35:49.372104 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:49.371884    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-metrics-certs podName:1a5f8182-95a6-4f4b-b40b-ea003c7be4dd nodeName:}" failed. No retries permitted until 2026-07-29 14:36:05.371876527 +0000 UTC m=+67.420334450 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-metrics-certs") pod "router-default-746f69dff9-2bdj7" (UID: "1a5f8182-95a6-4f4b-b40b-ea003c7be4dd") : secret "router-metrics-certs-default" not found
Jul 29 14:35:49.372104 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:49.371900    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-service-ca-bundle podName:1a5f8182-95a6-4f4b-b40b-ea003c7be4dd nodeName:}" failed. No retries permitted until 2026-07-29 14:36:05.3718923 +0000 UTC m=+67.420350217 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-service-ca-bundle") pod "router-default-746f69dff9-2bdj7" (UID: "1a5f8182-95a6-4f4b-b40b-ea003c7be4dd") : configmap references non-existent config key: service-ca.crt
Jul 29 14:35:49.372104 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:49.371915    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f010d9fc-7827-43c6-b9d8-03b9b09222ad-samples-operator-tls podName:f010d9fc-7827-43c6-b9d8-03b9b09222ad nodeName:}" failed. No retries permitted until 2026-07-29 14:36:05.371905905 +0000 UTC m=+67.420363821 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f010d9fc-7827-43c6-b9d8-03b9b09222ad-samples-operator-tls") pod "cluster-samples-operator-6748489656-67fqk" (UID: "f010d9fc-7827-43c6-b9d8-03b9b09222ad") : secret "samples-operator-tls" not found
Jul 29 14:35:52.402272 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:52.402237    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d86b4f92-11dc-402c-80cd-53bbb1981778-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mvd8l\" (UID: \"d86b4f92-11dc-402c-80cd-53bbb1981778\") " pod="openshift-insights/insights-runtime-extractor-mvd8l"
Jul 29 14:35:52.402763 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:52.402390    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d86b4f92-11dc-402c-80cd-53bbb1981778-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mvd8l\" (UID: \"d86b4f92-11dc-402c-80cd-53bbb1981778\") " pod="openshift-insights/insights-runtime-extractor-mvd8l"
Jul 29 14:35:52.402763 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:52.402470    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 14:35:52.402763 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:52.402537    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d86b4f92-11dc-402c-80cd-53bbb1981778-insights-runtime-extractor-tls podName:d86b4f92-11dc-402c-80cd-53bbb1981778 nodeName:}" failed. No retries permitted until 2026-07-29 14:36:00.402520509 +0000 UTC m=+62.450978427 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d86b4f92-11dc-402c-80cd-53bbb1981778-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mvd8l" (UID: "d86b4f92-11dc-402c-80cd-53bbb1981778") : secret "insights-runtime-extractor-tls" not found
Jul 29 14:35:52.402923 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:52.402876    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d86b4f92-11dc-402c-80cd-53bbb1981778-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mvd8l\" (UID: \"d86b4f92-11dc-402c-80cd-53bbb1981778\") " pod="openshift-insights/insights-runtime-extractor-mvd8l"
Jul 29 14:35:52.705858 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:52.705826    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6a44e5c-e648-4bb8-a393-7cfb4c22d761-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-qlc8b\" (UID: \"b6a44e5c-e648-4bb8-a393-7cfb4c22d761\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qlc8b"
Jul 29 14:35:52.706092 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:52.705975    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 14:35:52.706092 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:52.706047    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6a44e5c-e648-4bb8-a393-7cfb4c22d761-networking-console-plugin-cert podName:b6a44e5c-e648-4bb8-a393-7cfb4c22d761 nodeName:}" failed. No retries permitted until 2026-07-29 14:36:08.706031812 +0000 UTC m=+70.754489728 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b6a44e5c-e648-4bb8-a393-7cfb4c22d761-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-qlc8b" (UID: "b6a44e5c-e648-4bb8-a393-7cfb4c22d761") : secret "networking-console-plugin-cert" not found
Jul 29 14:35:53.660179 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:53.660110    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-xnm95"
Jul 29 14:35:53.660545 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:53.660200    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-xnm95"
Jul 29 14:35:53.660600 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:35:53.660549    2563 scope.go:117] "RemoveContainer" containerID="cf45f8fc57d0be003cd9ad212f6b72af584a8fedc2f87a05a7cd6af981bfbe4c"
Jul 29 14:35:53.660719 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:35:53.660700    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-xnm95_openshift-console-operator(23d79588-937d-4c6d-8760-a081024f608e)\"" pod="openshift-console-operator/console-operator-584dcb6b54-xnm95" podUID="23d79588-937d-4c6d-8760-a081024f608e"
Jul 29 14:36:00.475331 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:00.475291    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d86b4f92-11dc-402c-80cd-53bbb1981778-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mvd8l\" (UID: \"d86b4f92-11dc-402c-80cd-53bbb1981778\") " pod="openshift-insights/insights-runtime-extractor-mvd8l"
Jul 29 14:36:00.477516 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:00.477496    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d86b4f92-11dc-402c-80cd-53bbb1981778-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mvd8l\" (UID: \"d86b4f92-11dc-402c-80cd-53bbb1981778\") " pod="openshift-insights/insights-runtime-extractor-mvd8l"
Jul 29 14:36:00.567937 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:00.567904    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zhf56\""
Jul 29 14:36:00.576183 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:00.576162    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mvd8l"
Jul 29 14:36:00.642512 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:00.641686    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8z2d5"
Jul 29 14:36:00.728832 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:00.728758    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mvd8l"]
Jul 29 14:36:00.732641 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:36:00.732607    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd86b4f92_11dc_402c_80cd_53bbb1981778.slice/crio-6b3e142416bd4f2eeb73465d3c2d47722b2c40720bb5699b7939f210fad87695 WatchSource:0}: Error finding container 6b3e142416bd4f2eeb73465d3c2d47722b2c40720bb5699b7939f210fad87695: Status 404 returned error can't find the container with id 6b3e142416bd4f2eeb73465d3c2d47722b2c40720bb5699b7939f210fad87695
Jul 29 14:36:00.793687 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:00.793653    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mvd8l" event={"ID":"d86b4f92-11dc-402c-80cd-53bbb1981778","Type":"ContainerStarted","Data":"6b3e142416bd4f2eeb73465d3c2d47722b2c40720bb5699b7939f210fad87695"}
Jul 29 14:36:04.112069 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:04.112017    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95701def-c5e7-4309-a07b-6a7c05316dd6-original-pull-secret\") pod \"global-pull-secret-syncer-vg8f5\" (UID: \"95701def-c5e7-4309-a07b-6a7c05316dd6\") " pod="kube-system/global-pull-secret-syncer-vg8f5"
Jul 29 14:36:04.112542 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:04.112222    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad392a5c-af96-47d5-acdd-9905fc1cf0d5-metrics-certs\") pod \"network-metrics-daemon-vslsw\" (UID: \"ad392a5c-af96-47d5-acdd-9905fc1cf0d5\") " pod="openshift-multus/network-metrics-daemon-vslsw"
Jul 29 14:36:04.114322 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:04.114294    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 14:36:04.114634 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:04.114600    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 14:36:04.125036 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:04.124998    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad392a5c-af96-47d5-acdd-9905fc1cf0d5-metrics-certs\") pod \"network-metrics-daemon-vslsw\" (UID: \"ad392a5c-af96-47d5-acdd-9905fc1cf0d5\") " pod="openshift-multus/network-metrics-daemon-vslsw"
Jul 29 14:36:04.125898 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:04.125872    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95701def-c5e7-4309-a07b-6a7c05316dd6-original-pull-secret\") pod \"global-pull-secret-syncer-vg8f5\" (UID: \"95701def-c5e7-4309-a07b-6a7c05316dd6\") " pod="kube-system/global-pull-secret-syncer-vg8f5"
Jul 29 14:36:04.213568 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:04.213530    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wltn7\" (UniqueName: \"kubernetes.io/projected/e0828b12-3cef-4e42-8465-bf3dee289acb-kube-api-access-wltn7\") pod \"network-check-target-zvbtr\" (UID: \"e0828b12-3cef-4e42-8465-bf3dee289acb\") " pod="openshift-network-diagnostics/network-check-target-zvbtr"
Jul 29 14:36:04.216390 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:04.216366    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wltn7\" (UniqueName: \"kubernetes.io/projected/e0828b12-3cef-4e42-8465-bf3dee289acb-kube-api-access-wltn7\") pod \"network-check-target-zvbtr\" (UID: \"e0828b12-3cef-4e42-8465-bf3dee289acb\") " pod="openshift-network-diagnostics/network-check-target-zvbtr"
Jul 29 14:36:04.399469 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:04.399394    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ht8tz\""
Jul 29 14:36:04.407977 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:04.407946    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvbtr"
Jul 29 14:36:04.412374 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:04.412347    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vg8f5"
Jul 29 14:36:04.422001 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:04.421976    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nzccd\""
Jul 29 14:36:04.429869 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:04.429850    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vslsw"
Jul 29 14:36:05.322886 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:05.322851    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-registry-tls\") pod \"image-registry-5d8957c4d5-4pb5r\" (UID: \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\") " pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r"
Jul 29 14:36:05.325547 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:05.325520    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-registry-tls\") pod \"image-registry-5d8957c4d5-4pb5r\" (UID: \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\") " pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r"
Jul 29 14:36:05.424037 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:05.424002    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-metrics-certs\") pod \"router-default-746f69dff9-2bdj7\" (UID: \"1a5f8182-95a6-4f4b-b40b-ea003c7be4dd\") " pod="openshift-ingress/router-default-746f69dff9-2bdj7"
Jul 29 14:36:05.424037 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:05.424045    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-service-ca-bundle\") pod \"router-default-746f69dff9-2bdj7\" (UID: \"1a5f8182-95a6-4f4b-b40b-ea003c7be4dd\") " pod="openshift-ingress/router-default-746f69dff9-2bdj7"
Jul 29 14:36:05.424268 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:05.424242    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2037177-a695-47b3-bda0-0669bcfe408e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-bxjbs\" (UID: \"f2037177-a695-47b3-bda0-0669bcfe408e\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bxjbs"
Jul 29 14:36:05.424315 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:05.424294    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f010d9fc-7827-43c6-b9d8-03b9b09222ad-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-67fqk\" (UID: \"f010d9fc-7827-43c6-b9d8-03b9b09222ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-67fqk"
Jul 29 14:36:05.424364 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:05.424349    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4dbfdc61-952e-49ee-a2b2-a484d295b1e3-metrics-tls\") pod \"dns-default-4m4xj\" (UID: \"4dbfdc61-952e-49ee-a2b2-a484d295b1e3\") " pod="openshift-dns/dns-default-4m4xj"
Jul 29 14:36:05.424696 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:05.424667    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-service-ca-bundle\") pod \"router-default-746f69dff9-2bdj7\" (UID: \"1a5f8182-95a6-4f4b-b40b-ea003c7be4dd\") " pod="openshift-ingress/router-default-746f69dff9-2bdj7"
Jul 29 14:36:05.426468 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:05.426440    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a5f8182-95a6-4f4b-b40b-ea003c7be4dd-metrics-certs\") pod \"router-default-746f69dff9-2bdj7\" (UID: \"1a5f8182-95a6-4f4b-b40b-ea003c7be4dd\") " pod="openshift-ingress/router-default-746f69dff9-2bdj7"
Jul 29 14:36:05.426617 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:05.426600    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4dbfdc61-952e-49ee-a2b2-a484d295b1e3-metrics-tls\") pod \"dns-default-4m4xj\" (UID: \"4dbfdc61-952e-49ee-a2b2-a484d295b1e3\") " pod="openshift-dns/dns-default-4m4xj"
Jul 29 14:36:05.426677 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:05.426641    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2037177-a695-47b3-bda0-0669bcfe408e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-bxjbs\" (UID: \"f2037177-a695-47b3-bda0-0669bcfe408e\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bxjbs"
Jul 29 14:36:05.426997 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:05.426964    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f010d9fc-7827-43c6-b9d8-03b9b09222ad-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-67fqk\" (UID: \"f010d9fc-7827-43c6-b9d8-03b9b09222ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-67fqk"
Jul 29 14:36:05.476198 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:05.476167    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hfmq9\""
Jul 29 14:36:05.484766 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:05.484740    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r"
Jul 29 14:36:05.537686 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:05.537653    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8b7g5\""
Jul 29 14:36:05.546238 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:05.546204    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-746f69dff9-2bdj7"
Jul 29 14:36:05.559024 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:05.558996    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-tkzpb\""
Jul 29 14:36:05.567495 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:05.567469    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bxjbs"
Jul 29 14:36:05.575586 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:05.575524    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-g6266\""
Jul 29 14:36:05.583962 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:05.583938    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-67fqk"
Jul 29 14:36:05.640883 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:05.640852    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pp4lb\""
Jul 29 14:36:05.649536 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:05.649502    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4m4xj"
Jul 29 14:36:06.013631 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:06.013598    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vslsw"]
Jul 29 14:36:06.017895 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:36:06.017771    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad392a5c_af96_47d5_acdd_9905fc1cf0d5.slice/crio-a0bbb1dc35de5cd446119e667858a79279569917f652ff6532c6b3b68a4a0883 WatchSource:0}: Error finding container a0bbb1dc35de5cd446119e667858a79279569917f652ff6532c6b3b68a4a0883: Status 404 returned error can't find the container with id a0bbb1dc35de5cd446119e667858a79279569917f652ff6532c6b3b68a4a0883
Jul 29 14:36:06.045325 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:06.045285    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d8957c4d5-4pb5r"]
Jul 29 14:36:06.048624 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:36:06.048600    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51a472c7_19c4_4c5c_9813_ebf6cb7dd70c.slice/crio-542b73f077e2436bfacdcdce42fd835f3006e16c154fc3406f082186dabbcef1 WatchSource:0}: Error finding container 542b73f077e2436bfacdcdce42fd835f3006e16c154fc3406f082186dabbcef1: Status 404 returned error can't find the container with id 542b73f077e2436bfacdcdce42fd835f3006e16c154fc3406f082186dabbcef1
Jul 29 14:36:06.051668 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:06.051645    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-bxjbs"]
Jul 29 14:36:06.054259 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:36:06.054232    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2037177_a695_47b3_bda0_0669bcfe408e.slice/crio-875dd96dcb966b20e8522180b790a9c402012a9a8ede66902cfb96276e1ff2aa WatchSource:0}: Error finding container 875dd96dcb966b20e8522180b790a9c402012a9a8ede66902cfb96276e1ff2aa: Status 404 returned error can't find the container with id 875dd96dcb966b20e8522180b790a9c402012a9a8ede66902cfb96276e1ff2aa
Jul 29 14:36:06.275174 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:06.275124    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zvbtr"]
Jul 29 14:36:06.278617 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:06.278588    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4m4xj"]
Jul 29 14:36:06.279208 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:36:06.279052    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0828b12_3cef_4e42_8465_bf3dee289acb.slice/crio-7fcaa0d88863327148cf03df42778c6915cb90b974f2802e5172d6a0635639ff WatchSource:0}: Error finding container 7fcaa0d88863327148cf03df42778c6915cb90b974f2802e5172d6a0635639ff: Status 404 returned error can't find the container with id 7fcaa0d88863327148cf03df42778c6915cb90b974f2802e5172d6a0635639ff
Jul 29 14:36:06.279705 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:06.279668    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-67fqk"]
Jul 29 14:36:06.281300 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:36:06.281275    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dbfdc61_952e_49ee_a2b2_a484d295b1e3.slice/crio-51a2c6789d135f67ff34944fbb42a158b34cad0eba3bfdacf1b8bbf44d318f38 WatchSource:0}: Error finding container 51a2c6789d135f67ff34944fbb42a158b34cad0eba3bfdacf1b8bbf44d318f38: Status 404 returned error can't find the container with id 51a2c6789d135f67ff34944fbb42a158b34cad0eba3bfdacf1b8bbf44d318f38
Jul 29 14:36:06.318189 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:06.318161    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vg8f5"]
Jul 29 14:36:06.320973 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:06.320890    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-746f69dff9-2bdj7"]
Jul 29 14:36:06.321959 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:36:06.321932    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95701def_c5e7_4309_a07b_6a7c05316dd6.slice/crio-2ba478f970fd31a159dcb68f7250caa6c3776c6fc6decd9faf0969fa0d4de1bc WatchSource:0}: Error finding container 2ba478f970fd31a159dcb68f7250caa6c3776c6fc6decd9faf0969fa0d4de1bc: Status 404 returned error can't find the container with id 2ba478f970fd31a159dcb68f7250caa6c3776c6fc6decd9faf0969fa0d4de1bc
Jul 29 14:36:06.325017 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:36:06.324977    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a5f8182_95a6_4f4b_b40b_ea003c7be4dd.slice/crio-e7cc29e3a394569b8a1a1c586f403d016e41b1a835ddb055cbae20ff6d33b60b WatchSource:0}: Error finding container e7cc29e3a394569b8a1a1c586f403d016e41b1a835ddb055cbae20ff6d33b60b: Status 404 returned error can't find the container with id e7cc29e3a394569b8a1a1c586f403d016e41b1a835ddb055cbae20ff6d33b60b
Jul 29 14:36:06.820495 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:06.820456    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bxjbs" event={"ID":"f2037177-a695-47b3-bda0-0669bcfe408e","Type":"ContainerStarted","Data":"875dd96dcb966b20e8522180b790a9c402012a9a8ede66902cfb96276e1ff2aa"}
Jul 29 14:36:06.822075 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:06.822044    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-67fqk" event={"ID":"f010d9fc-7827-43c6-b9d8-03b9b09222ad","Type":"ContainerStarted","Data":"ca70e2b50f5b5bdda318b6d855a3218f21d651ca2b5de975496b79cf83460193"}
Jul 29 14:36:06.823948 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:06.823914    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mvd8l" event={"ID":"d86b4f92-11dc-402c-80cd-53bbb1981778","Type":"ContainerStarted","Data":"e5d37cfc61d8faed36255c97969c713dde8f33af807c4a76c6e76422e2365797"}
Jul 29 14:36:06.831145 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:06.831099    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r" event={"ID":"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c","Type":"ContainerStarted","Data":"5596ef7131279e5949c22653301cfd753bdc4d6eb2b6979b956955c10f058a63"}
Jul 29 14:36:06.831259 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:06.831199    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r" event={"ID":"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c","Type":"ContainerStarted","Data":"542b73f077e2436bfacdcdce42fd835f3006e16c154fc3406f082186dabbcef1"}
Jul 29 14:36:06.831437 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:06.831383    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r"
Jul 29 14:36:06.833289 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:06.833269    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-746f69dff9-2bdj7" event={"ID":"1a5f8182-95a6-4f4b-b40b-ea003c7be4dd","Type":"ContainerStarted","Data":"7a9246372404d8f3ee0304f332f74cbb5575b6002f88d4731580253a74bc9dc1"}
Jul 29 14:36:06.833408 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:06.833389    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-746f69dff9-2bdj7" event={"ID":"1a5f8182-95a6-4f4b-b40b-ea003c7be4dd","Type":"ContainerStarted","Data":"e7cc29e3a394569b8a1a1c586f403d016e41b1a835ddb055cbae20ff6d33b60b"}
Jul 29 14:36:06.835176 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:06.835154    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zvbtr" event={"ID":"e0828b12-3cef-4e42-8465-bf3dee289acb","Type":"ContainerStarted","Data":"ff000d4c05192b02db1ee7b4e09e8f7f45faf8bd04635eed1d31af1c276f2f6e"}
Jul 29 14:36:06.835276 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:06.835197    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zvbtr" event={"ID":"e0828b12-3cef-4e42-8465-bf3dee289acb","Type":"ContainerStarted","Data":"7fcaa0d88863327148cf03df42778c6915cb90b974f2802e5172d6a0635639ff"}
Jul 29 14:36:06.835276 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:06.835244    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zvbtr"
Jul 29 14:36:06.836617 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:06.836596    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vg8f5" event={"ID":"95701def-c5e7-4309-a07b-6a7c05316dd6","Type":"ContainerStarted","Data":"2ba478f970fd31a159dcb68f7250caa6c3776c6fc6decd9faf0969fa0d4de1bc"}
Jul 29 14:36:06.837932 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:06.837887    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vslsw" event={"ID":"ad392a5c-af96-47d5-acdd-9905fc1cf0d5","Type":"ContainerStarted","Data":"a0bbb1dc35de5cd446119e667858a79279569917f652ff6532c6b3b68a4a0883"}
Jul 29 14:36:06.839099 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:06.839068    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4m4xj" event={"ID":"4dbfdc61-952e-49ee-a2b2-a484d295b1e3","Type":"ContainerStarted","Data":"51a2c6789d135f67ff34944fbb42a158b34cad0eba3bfdacf1b8bbf44d318f38"}
Jul 29 14:36:06.852146 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:06.851878    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r" podStartSLOduration=67.851861739 podStartE2EDuration="1m7.851861739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:34:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:36:06.85064723 +0000 UTC m=+68.899105171" watchObservedRunningTime="2026-07-29 14:36:06.851861739 +0000 UTC m=+68.900319682"
Jul 29 14:36:06.869172 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:06.869107    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-746f69dff9-2bdj7" podStartSLOduration=55.869090829 podStartE2EDuration="55.869090829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:36:06.868538937 +0000 UTC m=+68.916996876" watchObservedRunningTime="2026-07-29 14:36:06.869090829 +0000 UTC m=+68.917548769"
Jul 29 14:36:06.882084 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:06.882036    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zvbtr" podStartSLOduration=67.882019244 podStartE2EDuration="1m7.882019244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:34:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:36:06.881069679 +0000 UTC m=+68.929527620" watchObservedRunningTime="2026-07-29 14:36:06.882019244 +0000 UTC m=+68.930477184"
Jul 29 14:36:07.547084 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:07.546847    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-746f69dff9-2bdj7"
Jul 29 14:36:07.549975 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:07.549945    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-746f69dff9-2bdj7"
Jul 29 14:36:07.843311 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:07.843240    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-746f69dff9-2bdj7"
Jul 29 14:36:07.844353 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:07.844336    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-746f69dff9-2bdj7"
Jul 29 14:36:08.391488 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.391459    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d74854d5b-h48n4"]
Jul 29 14:36:08.410384 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.410326    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb"]
Jul 29 14:36:08.414558 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.410462    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d74854d5b-h48n4"
Jul 29 14:36:08.418571 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.414851    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-hcph6\""
Jul 29 14:36:08.418571 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.415552    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 29 14:36:08.418571 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.415789    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 29 14:36:08.418571 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.416119    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 29 14:36:08.418571 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.416216    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 29 14:36:08.437231 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.436805    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7876954dd9-hxb5m"]
Jul 29 14:36:08.438148 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.437522    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb"
Jul 29 14:36:08.448356 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.447098    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 29 14:36:08.448356 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.447120    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 29 14:36:08.448356 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.447329    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 29 14:36:08.451275 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.450979    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 29 14:36:08.471861 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.471459    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7876954dd9-hxb5m"
Jul 29 14:36:08.474166 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.473961    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 29 14:36:08.479953 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.478668    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7876954dd9-hxb5m"]
Jul 29 14:36:08.479953 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.478714    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d74854d5b-h48n4"]
Jul 29 14:36:08.479953 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.478729    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb"]
Jul 29 14:36:08.479953 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.479146    2563 scope.go:117] "RemoveContainer" containerID="cf45f8fc57d0be003cd9ad212f6b72af584a8fedc2f87a05a7cd6af981bfbe4c"
Jul 29 14:36:08.554029 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.553994    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/25d4f56b-2265-4210-ba47-baf075294e43-klusterlet-config\") pod \"klusterlet-addon-workmgr-7876954dd9-hxb5m\" (UID: \"25d4f56b-2265-4210-ba47-baf075294e43\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7876954dd9-hxb5m"
Jul 29 14:36:08.554593 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.554046    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzmdq\" (UniqueName: \"kubernetes.io/projected/71854c3d-285d-4904-90fd-9907fb616fa5-kube-api-access-wzmdq\") pod \"managed-serviceaccount-addon-agent-5d74854d5b-h48n4\" (UID: \"71854c3d-285d-4904-90fd-9907fb616fa5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d74854d5b-h48n4"
Jul 29 14:36:08.554593 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.554192    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/78f16ce7-667e-40e9-a7a6-d68651f118de-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb\" (UID: \"78f16ce7-667e-40e9-a7a6-d68651f118de\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb"
Jul 29 14:36:08.554593 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.554240    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/78f16ce7-667e-40e9-a7a6-d68651f118de-hub\") pod \"cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb\" (UID: \"78f16ce7-667e-40e9-a7a6-d68651f118de\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb"
Jul 29 14:36:08.554593 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.554270    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78f16ce7-667e-40e9-a7a6-d68651f118de-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb\" (UID: \"78f16ce7-667e-40e9-a7a6-d68651f118de\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb"
Jul 29 14:36:08.554593 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.554315    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/78f16ce7-667e-40e9-a7a6-d68651f118de-ca\") pod \"cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb\" (UID: \"78f16ce7-667e-40e9-a7a6-d68651f118de\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb"
Jul 29 14:36:08.554593 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.554360    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/71854c3d-285d-4904-90fd-9907fb616fa5-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5d74854d5b-h48n4\" (UID: \"71854c3d-285d-4904-90fd-9907fb616fa5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d74854d5b-h48n4"
Jul 29 14:36:08.554593 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.554417    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25d4f56b-2265-4210-ba47-baf075294e43-tmp\") pod \"klusterlet-addon-workmgr-7876954dd9-hxb5m\" (UID: \"25d4f56b-2265-4210-ba47-baf075294e43\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7876954dd9-hxb5m"
Jul 29 14:36:08.554593 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.554468    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fspsr\" (UniqueName: \"kubernetes.io/projected/25d4f56b-2265-4210-ba47-baf075294e43-kube-api-access-fspsr\") pod \"klusterlet-addon-workmgr-7876954dd9-hxb5m\" (UID: \"25d4f56b-2265-4210-ba47-baf075294e43\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7876954dd9-hxb5m"
Jul 29 14:36:08.554593 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.554493    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f16ce7-667e-40e9-a7a6-d68651f118de-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb\" (UID: \"78f16ce7-667e-40e9-a7a6-d68651f118de\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb"
Jul 29 14:36:08.554593 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.554518    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl7gk\" (UniqueName: \"kubernetes.io/projected/78f16ce7-667e-40e9-a7a6-d68651f118de-kube-api-access-wl7gk\") pod \"cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb\" (UID: \"78f16ce7-667e-40e9-a7a6-d68651f118de\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb"
Jul 29 14:36:08.656034 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.654910    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/78f16ce7-667e-40e9-a7a6-d68651f118de-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb\" (UID: \"78f16ce7-667e-40e9-a7a6-d68651f118de\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb"
Jul 29 14:36:08.656034 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.654956    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/78f16ce7-667e-40e9-a7a6-d68651f118de-hub\") pod \"cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb\" (UID: \"78f16ce7-667e-40e9-a7a6-d68651f118de\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb"
Jul 29 14:36:08.656034 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.654988    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78f16ce7-667e-40e9-a7a6-d68651f118de-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb\" (UID: \"78f16ce7-667e-40e9-a7a6-d68651f118de\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb"
Jul 29 14:36:08.656034 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.655028    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/78f16ce7-667e-40e9-a7a6-d68651f118de-ca\") pod \"cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb\" (UID: \"78f16ce7-667e-40e9-a7a6-d68651f118de\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb"
Jul 29 14:36:08.656034 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.655056    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/71854c3d-285d-4904-90fd-9907fb616fa5-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5d74854d5b-h48n4\" (UID: \"71854c3d-285d-4904-90fd-9907fb616fa5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d74854d5b-h48n4"
Jul 29 14:36:08.656034 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.655095    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25d4f56b-2265-4210-ba47-baf075294e43-tmp\") pod \"klusterlet-addon-workmgr-7876954dd9-hxb5m\" (UID: \"25d4f56b-2265-4210-ba47-baf075294e43\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7876954dd9-hxb5m"
Jul 29 14:36:08.656034 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.655186    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fspsr\" (UniqueName: \"kubernetes.io/projected/25d4f56b-2265-4210-ba47-baf075294e43-kube-api-access-fspsr\") pod \"klusterlet-addon-workmgr-7876954dd9-hxb5m\" (UID: \"25d4f56b-2265-4210-ba47-baf075294e43\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7876954dd9-hxb5m"
Jul 29 14:36:08.656034 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.655214    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f16ce7-667e-40e9-a7a6-d68651f118de-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb\" (UID: \"78f16ce7-667e-40e9-a7a6-d68651f118de\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb"
Jul 29 14:36:08.656034 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.655240    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl7gk\" (UniqueName: \"kubernetes.io/projected/78f16ce7-667e-40e9-a7a6-d68651f118de-kube-api-access-wl7gk\") pod \"cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb\" (UID: \"78f16ce7-667e-40e9-a7a6-d68651f118de\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb"
Jul 29 14:36:08.656034 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.655276    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/25d4f56b-2265-4210-ba47-baf075294e43-klusterlet-config\") pod \"klusterlet-addon-workmgr-7876954dd9-hxb5m\" (UID: \"25d4f56b-2265-4210-ba47-baf075294e43\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7876954dd9-hxb5m"
Jul 29 14:36:08.656034 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.655316    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzmdq\" (UniqueName: \"kubernetes.io/projected/71854c3d-285d-4904-90fd-9907fb616fa5-kube-api-access-wzmdq\") pod \"managed-serviceaccount-addon-agent-5d74854d5b-h48n4\" (UID: \"71854c3d-285d-4904-90fd-9907fb616fa5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d74854d5b-h48n4"
Jul 29 14:36:08.657745 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.657055    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25d4f56b-2265-4210-ba47-baf075294e43-tmp\") pod \"klusterlet-addon-workmgr-7876954dd9-hxb5m\" (UID: \"25d4f56b-2265-4210-ba47-baf075294e43\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7876954dd9-hxb5m"
Jul 29 14:36:08.657917 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.657894    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f16ce7-667e-40e9-a7a6-d68651f118de-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb\" (UID: \"78f16ce7-667e-40e9-a7a6-d68651f118de\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb"
Jul 29 14:36:08.659499 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.659473    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/78f16ce7-667e-40e9-a7a6-d68651f118de-hub\") pod \"cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb\" (UID: \"78f16ce7-667e-40e9-a7a6-d68651f118de\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb"
Jul 29 14:36:08.661459 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.661407    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/78f16ce7-667e-40e9-a7a6-d68651f118de-ca\") pod \"cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb\" (UID: \"78f16ce7-667e-40e9-a7a6-d68651f118de\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb"
Jul 29 14:36:08.662456 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.662418    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78f16ce7-667e-40e9-a7a6-d68651f118de-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb\" (UID: \"78f16ce7-667e-40e9-a7a6-d68651f118de\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb"
Jul 29 14:36:08.663796 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.663034    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/25d4f56b-2265-4210-ba47-baf075294e43-klusterlet-config\") pod \"klusterlet-addon-workmgr-7876954dd9-hxb5m\" (UID: \"25d4f56b-2265-4210-ba47-baf075294e43\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7876954dd9-hxb5m"
Jul 29 14:36:08.663796 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.663450    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/78f16ce7-667e-40e9-a7a6-d68651f118de-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb\" (UID: \"78f16ce7-667e-40e9-a7a6-d68651f118de\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb"
Jul 29 14:36:08.663944 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.663793    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/71854c3d-285d-4904-90fd-9907fb616fa5-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5d74854d5b-h48n4\" (UID: \"71854c3d-285d-4904-90fd-9907fb616fa5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d74854d5b-h48n4"
Jul 29 14:36:08.666380 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.666356    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl7gk\" (UniqueName: \"kubernetes.io/projected/78f16ce7-667e-40e9-a7a6-d68651f118de-kube-api-access-wl7gk\") pod \"cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb\" (UID: \"78f16ce7-667e-40e9-a7a6-d68651f118de\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb"
Jul 29 14:36:08.667395 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.667165    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fspsr\" (UniqueName: \"kubernetes.io/projected/25d4f56b-2265-4210-ba47-baf075294e43-kube-api-access-fspsr\") pod \"klusterlet-addon-workmgr-7876954dd9-hxb5m\" (UID: \"25d4f56b-2265-4210-ba47-baf075294e43\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7876954dd9-hxb5m"
Jul 29 14:36:08.667518 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.667417    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzmdq\" (UniqueName: \"kubernetes.io/projected/71854c3d-285d-4904-90fd-9907fb616fa5-kube-api-access-wzmdq\") pod \"managed-serviceaccount-addon-agent-5d74854d5b-h48n4\" (UID: \"71854c3d-285d-4904-90fd-9907fb616fa5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d74854d5b-h48n4"
Jul 29 14:36:08.739901 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.739322    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d74854d5b-h48n4"
Jul 29 14:36:08.750431 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.750376    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb"
Jul 29 14:36:08.756556 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.756516    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6a44e5c-e648-4bb8-a393-7cfb4c22d761-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-qlc8b\" (UID: \"b6a44e5c-e648-4bb8-a393-7cfb4c22d761\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qlc8b"
Jul 29 14:36:08.759831 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.759804    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6a44e5c-e648-4bb8-a393-7cfb4c22d761-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-qlc8b\" (UID: \"b6a44e5c-e648-4bb8-a393-7cfb4c22d761\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qlc8b"
Jul 29 14:36:08.792208 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.792072    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7876954dd9-hxb5m"
Jul 29 14:36:08.922387 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.922298    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-tzdkm\""
Jul 29 14:36:08.930095 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:08.930056    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-qlc8b"
Jul 29 14:36:12.661174 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:12.661103    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7876954dd9-hxb5m"]
Jul 29 14:36:14.187733 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:36:14.187685    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25d4f56b_2265_4210_ba47_baf075294e43.slice/crio-d71a14d1caeb287ff14a31ae6506d5a38a996f23e73f76254e69302f92f14ebb WatchSource:0}: Error finding container d71a14d1caeb287ff14a31ae6506d5a38a996f23e73f76254e69302f92f14ebb: Status 404 returned error can't find the container with id d71a14d1caeb287ff14a31ae6506d5a38a996f23e73f76254e69302f92f14ebb
Jul 29 14:36:14.341039 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:14.339874    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d74854d5b-h48n4"]
Jul 29 14:36:14.362051 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:14.361993    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb"]
Jul 29 14:36:14.375038 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:14.375013    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-qlc8b"]
Jul 29 14:36:14.406458 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:36:14.406421    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71854c3d_285d_4904_90fd_9907fb616fa5.slice/crio-7fc22282284c24b2fa0d4819ede8bca5a77dfae72f20c8ea7b3ca9c75124b853 WatchSource:0}: Error finding container 7fc22282284c24b2fa0d4819ede8bca5a77dfae72f20c8ea7b3ca9c75124b853: Status 404 returned error can't find the container with id 7fc22282284c24b2fa0d4819ede8bca5a77dfae72f20c8ea7b3ca9c75124b853
Jul 29 14:36:14.407108 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:36:14.406935    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78f16ce7_667e_40e9_a7a6_d68651f118de.slice/crio-83b79babd92005c33bb281e26b72327ce57cd37d996fef1a75bac796801b50fb WatchSource:0}: Error finding container 83b79babd92005c33bb281e26b72327ce57cd37d996fef1a75bac796801b50fb: Status 404 returned error can't find the container with id 83b79babd92005c33bb281e26b72327ce57cd37d996fef1a75bac796801b50fb
Jul 29 14:36:14.408248 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:36:14.408225    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a44e5c_e648_4bb8_a393_7cfb4c22d761.slice/crio-01331fdd3ce78ebcc97493ee68a0bd273cc7a5b7591b42ab7695f83483a6d682 WatchSource:0}: Error finding container 01331fdd3ce78ebcc97493ee68a0bd273cc7a5b7591b42ab7695f83483a6d682: Status 404 returned error can't find the container with id 01331fdd3ce78ebcc97493ee68a0bd273cc7a5b7591b42ab7695f83483a6d682
Jul 29 14:36:14.891648 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:14.891561    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-67fqk" event={"ID":"f010d9fc-7827-43c6-b9d8-03b9b09222ad","Type":"ContainerStarted","Data":"27b91eb4e6e2f800e11ef8cf3b125eed10de8c30a70189bbf0794ce58c8b509e"}
Jul 29 14:36:14.895209 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:14.895121    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mvd8l" event={"ID":"d86b4f92-11dc-402c-80cd-53bbb1981778","Type":"ContainerStarted","Data":"c0186c6c50307f8e5a6c02b94fef007c4dcedc424e13d2cf64cc270a664f2247"}
Jul 29 14:36:14.898078 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:14.898004    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vg8f5" event={"ID":"95701def-c5e7-4309-a07b-6a7c05316dd6","Type":"ContainerStarted","Data":"0541d22dbdc725f7da212e3e7406d8ceb56285003eabb3bcc903980685e7efd9"}
Jul 29 14:36:14.904891 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:14.904363    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vslsw" event={"ID":"ad392a5c-af96-47d5-acdd-9905fc1cf0d5","Type":"ContainerStarted","Data":"58a6de8f954fa60a90acbe18e353b4f54a1c0d146cbd6ebc3024822c698a3788"}
Jul 29 14:36:14.904891 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:14.904410    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vslsw" event={"ID":"ad392a5c-af96-47d5-acdd-9905fc1cf0d5","Type":"ContainerStarted","Data":"e635000a5abd36035b99aa0018f436e2b2e7603cac482181ff1964bb19d9f123"}
Jul 29 14:36:14.908599 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:14.908543    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4m4xj" event={"ID":"4dbfdc61-952e-49ee-a2b2-a484d295b1e3","Type":"ContainerStarted","Data":"5fd9306121a83b57581fcaedce522bfde78f5b13d4af1cc69e0dff9dbb00208b"}
Jul 29 14:36:14.911058 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:14.910850    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xnm95_23d79588-937d-4c6d-8760-a081024f608e/console-operator/1.log"
Jul 29 14:36:14.911058 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:14.910922    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-xnm95" event={"ID":"23d79588-937d-4c6d-8760-a081024f608e","Type":"ContainerStarted","Data":"b3afc72c4ef66803b7f178f11939e180e23d260df27d44de9cbd727d61225a72"}
Jul 29 14:36:14.911790 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:14.911739    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-xnm95"
Jul 29 14:36:14.914090 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:14.913589    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vg8f5" podStartSLOduration=67.590580951 podStartE2EDuration="1m15.913571717s" totalImagesPullingTime="8.322990766s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 14:34:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:36:14.911530153 +0000 UTC m=+76.959988093" watchObservedRunningTime="2026-07-29 14:36:14.913571717 +0000 UTC m=+76.962029654"
Jul 29 14:36:14.916248 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:14.916221    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bxjbs" event={"ID":"f2037177-a695-47b3-bda0-0669bcfe408e","Type":"ContainerStarted","Data":"66b3f9d1fa931244e49b0e967471063224b099be0a82fd5780339be80253e80f"}
Jul 29 14:36:14.918273 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:14.918249    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb" event={"ID":"78f16ce7-667e-40e9-a7a6-d68651f118de","Type":"ContainerStarted","Data":"83b79babd92005c33bb281e26b72327ce57cd37d996fef1a75bac796801b50fb"}
Jul 29 14:36:14.920844 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:14.920816    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-qlc8b" event={"ID":"b6a44e5c-e648-4bb8-a393-7cfb4c22d761","Type":"ContainerStarted","Data":"01331fdd3ce78ebcc97493ee68a0bd273cc7a5b7591b42ab7695f83483a6d682"}
Jul 29 14:36:14.922213 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:14.922190    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d74854d5b-h48n4" event={"ID":"71854c3d-285d-4904-90fd-9907fb616fa5","Type":"ContainerStarted","Data":"7fc22282284c24b2fa0d4819ede8bca5a77dfae72f20c8ea7b3ca9c75124b853"}
Jul 29 14:36:14.923604 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:14.923562    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7876954dd9-hxb5m" event={"ID":"25d4f56b-2265-4210-ba47-baf075294e43","Type":"ContainerStarted","Data":"d71a14d1caeb287ff14a31ae6506d5a38a996f23e73f76254e69302f92f14ebb"}
Jul 29 14:36:14.927806 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:14.926885    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vslsw" podStartSLOduration=70.831215277 podStartE2EDuration="1m15.926868526s" totalImagesPullingTime="5.095653249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:34:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:36:14.924941249 +0000 UTC m=+76.973399188" watchObservedRunningTime="2026-07-29 14:36:14.926868526 +0000 UTC m=+76.975326466"
Jul 29 14:36:14.943002 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:14.942889    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-xnm95" podStartSLOduration=54.341182109 podStartE2EDuration="1m3.942872911s" totalImagesPullingTime="9.601690802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:35:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:36:14.942872131 +0000 UTC m=+76.991330071" watchObservedRunningTime="2026-07-29 14:36:14.942872911 +0000 UTC m=+76.991330853"
Jul 29 14:36:14.960121 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:14.959649    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bxjbs" podStartSLOduration=55.828032363 podStartE2EDuration="1m3.959629897s" totalImagesPullingTime="8.131597534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:35:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:36:14.957029342 +0000 UTC m=+77.005487280" watchObservedRunningTime="2026-07-29 14:36:14.959629897 +0000 UTC m=+77.008087833"
Jul 29 14:36:15.262432 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:15.262401    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-xnm95"
Jul 29 14:36:15.932095 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:15.931277    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-67fqk" event={"ID":"f010d9fc-7827-43c6-b9d8-03b9b09222ad","Type":"ContainerStarted","Data":"3b9a1cf5c2f2100c6d644ba2b2aae7e293d68cfd80b0e33733144035af5d9a0d"}
Jul 29 14:36:15.935898 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:15.935866    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4m4xj" event={"ID":"4dbfdc61-952e-49ee-a2b2-a484d295b1e3","Type":"ContainerStarted","Data":"c78d385058663473c7710629a8e4ab95631356fc05a32c901ee42f023081a9b2"}
Jul 29 14:36:15.949859 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:15.949306    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-67fqk" podStartSLOduration=57.092300589 podStartE2EDuration="1m4.949285727s" totalImagesPullingTime="7.856985138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:35:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:36:15.947637274 +0000 UTC m=+77.996095216" watchObservedRunningTime="2026-07-29 14:36:15.949285727 +0000 UTC m=+77.997743667"
Jul 29 14:36:15.969406 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:15.969218    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4m4xj" podStartSLOduration=35.064129107 podStartE2EDuration="42.969196822s" totalImagesPullingTime="7.905067715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:35:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:36:15.967401467 +0000 UTC m=+78.015859407" watchObservedRunningTime="2026-07-29 14:36:15.969196822 +0000 UTC m=+78.017654764"
Jul 29 14:36:16.944126 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:16.943952    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4m4xj"
Jul 29 14:36:22.974725 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:22.974686    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-qlc8b" event={"ID":"b6a44e5c-e648-4bb8-a393-7cfb4c22d761","Type":"ContainerStarted","Data":"8b89919391b8851d0332c1387f0b20f262a0265b4d72d84fc2be24bda134bcdf"}
Jul 29 14:36:22.976215 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:22.976189    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d74854d5b-h48n4" event={"ID":"71854c3d-285d-4904-90fd-9907fb616fa5","Type":"ContainerStarted","Data":"a80baab8a20142d60d13353a54e91bf9cdd925f94200e91b248e00f47a892e3d"}
Jul 29 14:36:22.977548 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:22.977525    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7876954dd9-hxb5m" event={"ID":"25d4f56b-2265-4210-ba47-baf075294e43","Type":"ContainerStarted","Data":"b4d3f24bfbaab1948d100055acd8b7f2fcc786b6aff5c4c43c0016ec4ba383b3"}
Jul 29 14:36:22.977682 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:22.977665    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7876954dd9-hxb5m"
Jul 29 14:36:22.979593 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:22.979574    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7876954dd9-hxb5m"
Jul 29 14:36:22.979742 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:22.979724    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mvd8l" event={"ID":"d86b4f92-11dc-402c-80cd-53bbb1981778","Type":"ContainerStarted","Data":"47498e4cc6b1ad43be120cea9c40609d138f4d7de13c5cd92bc251de8a611f1b"}
Jul 29 14:36:22.981610 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:22.981588    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb" event={"ID":"78f16ce7-667e-40e9-a7a6-d68651f118de","Type":"ContainerStarted","Data":"4ef6356a5b9811370fb5d399908a4f66cb45c3b441e13a0788715f135cded197"}
Jul 29 14:36:22.981695 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:22.981622    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb" event={"ID":"78f16ce7-667e-40e9-a7a6-d68651f118de","Type":"ContainerStarted","Data":"d330b95962f71c316540386975c4fcee64fae496722288bf73655ddd00e5fd18"}
Jul 29 14:36:22.981695 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:22.981640    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb" event={"ID":"78f16ce7-667e-40e9-a7a6-d68651f118de","Type":"ContainerStarted","Data":"e79a90c525901e39e7706444e9049e0ba477b3c92fce46949293defc30fb372e"}
Jul 29 14:36:22.992108 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:22.992066    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-qlc8b" podStartSLOduration=39.669390947 podStartE2EDuration="46.992054406s" totalImagesPullingTime="7.322663459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:35:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:36:22.990932488 +0000 UTC m=+85.039390426" watchObservedRunningTime="2026-07-29 14:36:22.992054406 +0000 UTC m=+85.040512344"
Jul 29 14:36:23.035171 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:23.035098    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d74854d5b-h48n4" podStartSLOduration=7.240174506 podStartE2EDuration="15.035076251s" totalImagesPullingTime="7.794901745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:36:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:36:23.034811094 +0000 UTC m=+85.083269033" watchObservedRunningTime="2026-07-29 14:36:23.035076251 +0000 UTC m=+85.083534191"
Jul 29 14:36:23.035320 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:23.035206    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mvd8l" podStartSLOduration=18.112551656 podStartE2EDuration="39.03519938s" totalImagesPullingTime="20.922647724s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 14:35:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:36:23.016201834 +0000 UTC m=+85.064659775" watchObservedRunningTime="2026-07-29 14:36:23.03519938 +0000 UTC m=+85.083657319"
Jul 29 14:36:23.057186 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:23.057115    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7876954dd9-hxb5m" podStartSLOduration=7.314501477 podStartE2EDuration="15.05709839s" totalImagesPullingTime="7.742596913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:36:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:36:23.055984782 +0000 UTC m=+85.104442723" watchObservedRunningTime="2026-07-29 14:36:23.05709839 +0000 UTC m=+85.105556329"
Jul 29 14:36:23.076394 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:23.076352    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7fcf6fb7-nt8pb" podStartSLOduration=7.285227749 podStartE2EDuration="15.076338766s" totalImagesPullingTime="7.791111017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:36:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:36:23.075733106 +0000 UTC m=+85.124191046" watchObservedRunningTime="2026-07-29 14:36:23.076338766 +0000 UTC m=+85.124796704"
Jul 29 14:36:25.489682 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:25.489645    2563 patch_prober.go:28] interesting pod/image-registry-5d8957c4d5-4pb5r container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 29 14:36:25.490069 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:25.489702    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r" podUID="51a472c7-19c4-4c5c-9813-ebf6cb7dd70c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 14:36:26.950624 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:26.950594    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4m4xj"
Jul 29 14:36:27.848497 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:27.848454    2563 patch_prober.go:28] interesting pod/image-registry-5d8957c4d5-4pb5r container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 29 14:36:27.848779 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:27.848748    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r" podUID="51a472c7-19c4-4c5c-9813-ebf6cb7dd70c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 14:36:30.739825 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:30.739781    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8zfjf"]
Jul 29 14:36:30.743481 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:30.743457    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8zfjf"
Jul 29 14:36:30.747195 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:30.746896    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 29 14:36:30.747195 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:30.746908    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 29 14:36:30.747195 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:30.747020    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 29 14:36:30.747195 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:30.747187    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 29 14:36:30.747793 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:30.747768    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5nz5x\""
Jul 29 14:36:30.820600 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:30.820567    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85-metrics-client-ca\") pod \"node-exporter-8zfjf\" (UID: \"8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85\") " pod="openshift-monitoring/node-exporter-8zfjf"
Jul 29 14:36:30.820600 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:30.820604    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85-node-exporter-wtmp\") pod \"node-exporter-8zfjf\" (UID: \"8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85\") " pod="openshift-monitoring/node-exporter-8zfjf"
Jul 29 14:36:30.820815 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:30.820628    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85-root\") pod \"node-exporter-8zfjf\" (UID: \"8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85\") " pod="openshift-monitoring/node-exporter-8zfjf"
Jul 29 14:36:30.820815 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:30.820713    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85-node-exporter-textfile\") pod \"node-exporter-8zfjf\" (UID: \"8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85\") " pod="openshift-monitoring/node-exporter-8zfjf"
Jul 29 14:36:30.820815 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:30.820762    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8zfjf\" (UID: \"8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85\") " pod="openshift-monitoring/node-exporter-8zfjf"
Jul 29 14:36:30.820918 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:30.820823    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8dc7\" (UniqueName: \"kubernetes.io/projected/8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85-kube-api-access-s8dc7\") pod \"node-exporter-8zfjf\" (UID: \"8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85\") " pod="openshift-monitoring/node-exporter-8zfjf"
Jul 29 14:36:30.820918 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:30.820845    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85-node-exporter-tls\") pod \"node-exporter-8zfjf\" (UID: \"8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85\") " pod="openshift-monitoring/node-exporter-8zfjf"
Jul 29 14:36:30.820918 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:30.820874    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85-node-exporter-accelerators-collector-config\") pod \"node-exporter-8zfjf\" (UID: \"8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85\") " pod="openshift-monitoring/node-exporter-8zfjf"
Jul 29 14:36:30.820918 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:30.820903    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85-sys\") pod \"node-exporter-8zfjf\" (UID: \"8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85\") " pod="openshift-monitoring/node-exporter-8zfjf"
Jul 29 14:36:30.921340 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:30.921308    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8zfjf\" (UID: \"8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85\") " pod="openshift-monitoring/node-exporter-8zfjf"
Jul 29 14:36:30.921514 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:30.921349    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8dc7\" (UniqueName: \"kubernetes.io/projected/8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85-kube-api-access-s8dc7\") pod \"node-exporter-8zfjf\" (UID: \"8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85\") " pod="openshift-monitoring/node-exporter-8zfjf"
Jul 29 14:36:30.921514 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:30.921373    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85-node-exporter-tls\") pod \"node-exporter-8zfjf\" (UID: \"8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85\") " pod="openshift-monitoring/node-exporter-8zfjf"
Jul 29 14:36:30.921514 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:30.921396    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85-node-exporter-accelerators-collector-config\") pod \"node-exporter-8zfjf\" (UID: \"8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85\") " pod="openshift-monitoring/node-exporter-8zfjf"
Jul 29 14:36:30.921514 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:36:30.921492    2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 29 14:36:30.921726 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:36:30.921567    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85-node-exporter-tls podName:8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85 nodeName:}" failed. No retries permitted until 2026-07-29 14:36:31.421544011 +0000 UTC m=+93.470001931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85-node-exporter-tls") pod "node-exporter-8zfjf" (UID: "8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85") : secret "node-exporter-tls" not found
Jul 29 14:36:30.921726 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:30.921590    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85-sys\") pod \"node-exporter-8zfjf\" (UID: \"8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85\") " pod="openshift-monitoring/node-exporter-8zfjf"
Jul 29 14:36:30.921726 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:30.921698    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85-metrics-client-ca\") pod \"node-exporter-8zfjf\" (UID: \"8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85\") " pod="openshift-monitoring/node-exporter-8zfjf"
Jul 29 14:36:30.921885 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:30.921732    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85-node-exporter-wtmp\") pod \"node-exporter-8zfjf\" (UID: \"8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85\") " pod="openshift-monitoring/node-exporter-8zfjf"
Jul 29 14:36:30.921885 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:30.921729    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85-sys\") pod \"node-exporter-8zfjf\" (UID: \"8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85\") " pod="openshift-monitoring/node-exporter-8zfjf"
Jul 29 14:36:30.921885 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:30.921767    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85-root\") pod \"node-exporter-8zfjf\" (UID: \"8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85\") " pod="openshift-monitoring/node-exporter-8zfjf"
Jul 29 14:36:30.921997 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:30.921877    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85-node-exporter-wtmp\") pod \"node-exporter-8zfjf\" (UID: \"8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85\") " pod="openshift-monitoring/node-exporter-8zfjf"
Jul 29 14:36:30.921997 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:30.921885    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85-node-exporter-textfile\") pod \"node-exporter-8zfjf\" (UID: \"8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85\") " pod="openshift-monitoring/node-exporter-8zfjf"
Jul 29 14:36:30.921997 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:30.921804    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85-root\") pod \"node-exporter-8zfjf\" (UID: \"8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85\") " pod="openshift-monitoring/node-exporter-8zfjf"
Jul 29 14:36:30.922086 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:30.922009    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85-node-exporter-accelerators-collector-config\") pod \"node-exporter-8zfjf\" (UID: \"8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85\") " pod="openshift-monitoring/node-exporter-8zfjf"
Jul 29 14:36:30.922145 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:30.922114    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85-node-exporter-textfile\") pod \"node-exporter-8zfjf\" (UID: \"8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85\") " pod="openshift-monitoring/node-exporter-8zfjf"
Jul 29 14:36:30.922295 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:30.922277    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85-metrics-client-ca\") pod \"node-exporter-8zfjf\" (UID: \"8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85\") " pod="openshift-monitoring/node-exporter-8zfjf"
Jul 29 14:36:30.923738 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:30.923718    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8zfjf\" (UID: \"8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85\") " pod="openshift-monitoring/node-exporter-8zfjf"
Jul 29 14:36:30.936765 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:30.936737    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8dc7\" (UniqueName: \"kubernetes.io/projected/8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85-kube-api-access-s8dc7\") pod \"node-exporter-8zfjf\" (UID: \"8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85\") " pod="openshift-monitoring/node-exporter-8zfjf"
Jul 29 14:36:31.425592 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:31.425555    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85-node-exporter-tls\") pod \"node-exporter-8zfjf\" (UID: \"8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85\") " pod="openshift-monitoring/node-exporter-8zfjf"
Jul 29 14:36:31.427931 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:31.427906    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85-node-exporter-tls\") pod \"node-exporter-8zfjf\" (UID: \"8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85\") " pod="openshift-monitoring/node-exporter-8zfjf"
Jul 29 14:36:31.656004 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:31.655967    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8zfjf"
Jul 29 14:36:31.667696 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:36:31.667669    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c0a2c1f_1b0d_4f90_bb78_dd2ee6dede85.slice/crio-73288cf8fb79c4bcd2407a093fb05882371a4cc82798bc38dc2346eff06a20cd WatchSource:0}: Error finding container 73288cf8fb79c4bcd2407a093fb05882371a4cc82798bc38dc2346eff06a20cd: Status 404 returned error can't find the container with id 73288cf8fb79c4bcd2407a093fb05882371a4cc82798bc38dc2346eff06a20cd
Jul 29 14:36:32.016981 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:32.016942    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8zfjf" event={"ID":"8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85","Type":"ContainerStarted","Data":"73288cf8fb79c4bcd2407a093fb05882371a4cc82798bc38dc2346eff06a20cd"}
Jul 29 14:36:33.021933 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:33.021897    2563 generic.go:356] "Generic (PLEG): container finished" podID="8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85" containerID="697791e6cfad89565f7ad0ea1ec410056739a86f83a462345c622d772b5ff923" exitCode=0
Jul 29 14:36:33.022398 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:33.021978    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8zfjf" event={"ID":"8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85","Type":"ContainerDied","Data":"697791e6cfad89565f7ad0ea1ec410056739a86f83a462345c622d772b5ff923"}
Jul 29 14:36:34.028306 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:34.028271    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8zfjf" event={"ID":"8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85","Type":"ContainerStarted","Data":"82a5358341b25ac3511a9cc5fc61d1ce6e3596ce5bdf7f2502f86bb69c7f3f35"}
Jul 29 14:36:34.028676 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:34.028325    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8zfjf" event={"ID":"8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85","Type":"ContainerStarted","Data":"c3ed3055edafd90ad6a9c10354914ca76885cedd10e108b18933715def7f5ee0"}
Jul 29 14:36:34.057373 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:34.057321    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8zfjf" podStartSLOduration=2.998260472 podStartE2EDuration="4.057303619s" totalImagesPullingTime="993.451892ms" totalInitContainerRuntime="65.591255ms" isStatefulPod=true podCreationTimestamp="2026-07-29 14:36:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:36:34.053665692 +0000 UTC m=+96.102123632" watchObservedRunningTime="2026-07-29 14:36:34.057303619 +0000 UTC m=+96.105761557"
Jul 29 14:36:35.072329 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:35.072294    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d8957c4d5-4pb5r"]
Jul 29 14:36:35.078160 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:35.078112    2563 patch_prober.go:28] interesting pod/image-registry-5d8957c4d5-4pb5r container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 29 14:36:35.078289 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:35.078176    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r" podUID="51a472c7-19c4-4c5c-9813-ebf6cb7dd70c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 14:36:37.846232 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:37.846202    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zvbtr"
Jul 29 14:36:45.081367 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:45.081336    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r"
Jul 29 14:36:50.095525 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:50.095490    2563 generic.go:356] "Generic (PLEG): container finished" podID="cca0fc93-ab3f-4cb2-aaab-8fcdc4428636" containerID="465d70de46cffa037ec1869b7184acd88cdb73aea39404ec32cc60a93b09867e" exitCode=0
Jul 29 14:36:50.095944 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:50.095540    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9j4jg" event={"ID":"cca0fc93-ab3f-4cb2-aaab-8fcdc4428636","Type":"ContainerDied","Data":"465d70de46cffa037ec1869b7184acd88cdb73aea39404ec32cc60a93b09867e"}
Jul 29 14:36:50.095944 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:50.095827    2563 scope.go:117] "RemoveContainer" containerID="465d70de46cffa037ec1869b7184acd88cdb73aea39404ec32cc60a93b09867e"
Jul 29 14:36:51.101956 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:51.101921    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9j4jg" event={"ID":"cca0fc93-ab3f-4cb2-aaab-8fcdc4428636","Type":"ContainerStarted","Data":"b1180732190f8464315e1e093f1b453be6650113b28d0e0bf808e4286196604f"}
Jul 29 14:36:55.118003 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:55.117920    2563 generic.go:356] "Generic (PLEG): container finished" podID="7504b97a-38e2-44ff-822a-a7f1e98b8933" containerID="417e1d881d2eb45cb8401f8d06b7a2a521a365994b4cf1120efec88221eb8243" exitCode=0
Jul 29 14:36:55.118003 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:55.117992    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-fkjfz" event={"ID":"7504b97a-38e2-44ff-822a-a7f1e98b8933","Type":"ContainerDied","Data":"417e1d881d2eb45cb8401f8d06b7a2a521a365994b4cf1120efec88221eb8243"}
Jul 29 14:36:55.118480 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:55.118308    2563 scope.go:117] "RemoveContainer" containerID="417e1d881d2eb45cb8401f8d06b7a2a521a365994b4cf1120efec88221eb8243"
Jul 29 14:36:56.130801 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:36:56.130761    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-fkjfz" event={"ID":"7504b97a-38e2-44ff-822a-a7f1e98b8933","Type":"ContainerStarted","Data":"93c1fd3b49a7b93a6d9fff691289917406697727ab0131b81b8b8851b5ca9c37"}
Jul 29 14:37:00.092449 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:37:00.092384    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r" podUID="51a472c7-19c4-4c5c-9813-ebf6cb7dd70c" containerName="registry" containerID="cri-o://5596ef7131279e5949c22653301cfd753bdc4d6eb2b6979b956955c10f058a63" gracePeriod=30
Jul 29 14:37:00.342585 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:37:00.342527    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r"
Jul 29 14:37:00.464149 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:37:00.464108    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-bound-sa-token\") pod \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\" (UID: \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\") "
Jul 29 14:37:00.464326 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:37:00.464167    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-registry-certificates\") pod \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\" (UID: \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\") "
Jul 29 14:37:00.464326 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:37:00.464196    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jsccx\" (UniqueName: \"kubernetes.io/projected/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-kube-api-access-jsccx\") pod \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\" (UID: \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\") "
Jul 29 14:37:00.464326 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:37:00.464240    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-registry-tls\") pod \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\" (UID: \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\") "
Jul 29 14:37:00.464326 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:37:00.464275    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-image-registry-private-configuration\") pod \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\" (UID: \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\") "
Jul 29 14:37:00.464326 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:37:00.464301    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-installation-pull-secrets\") pod \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\" (UID: \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\") "
Jul 29 14:37:00.464564 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:37:00.464360    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-ca-trust-extracted\") pod \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\" (UID: \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\") "
Jul 29 14:37:00.464564 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:37:00.464385    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-trusted-ca\") pod \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\" (UID: \"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c\") "
Jul 29 14:37:00.465628 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:37:00.464735    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "51a472c7-19c4-4c5c-9813-ebf6cb7dd70c" (UID: "51a472c7-19c4-4c5c-9813-ebf6cb7dd70c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 14:37:00.465628 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:37:00.464983    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "51a472c7-19c4-4c5c-9813-ebf6cb7dd70c" (UID: "51a472c7-19c4-4c5c-9813-ebf6cb7dd70c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 14:37:00.466777 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:37:00.466747    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-kube-api-access-jsccx" (OuterVolumeSpecName: "kube-api-access-jsccx") pod "51a472c7-19c4-4c5c-9813-ebf6cb7dd70c" (UID: "51a472c7-19c4-4c5c-9813-ebf6cb7dd70c"). InnerVolumeSpecName "kube-api-access-jsccx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 14:37:00.466909 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:37:00.466804    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "51a472c7-19c4-4c5c-9813-ebf6cb7dd70c" (UID: "51a472c7-19c4-4c5c-9813-ebf6cb7dd70c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 14:37:00.466909 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:37:00.466866    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "51a472c7-19c4-4c5c-9813-ebf6cb7dd70c" (UID: "51a472c7-19c4-4c5c-9813-ebf6cb7dd70c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 14:37:00.467181 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:37:00.467161    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "51a472c7-19c4-4c5c-9813-ebf6cb7dd70c" (UID: "51a472c7-19c4-4c5c-9813-ebf6cb7dd70c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 14:37:00.467274 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:37:00.467248    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "51a472c7-19c4-4c5c-9813-ebf6cb7dd70c" (UID: "51a472c7-19c4-4c5c-9813-ebf6cb7dd70c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 14:37:00.472816 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:37:00.472791    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "51a472c7-19c4-4c5c-9813-ebf6cb7dd70c" (UID: "51a472c7-19c4-4c5c-9813-ebf6cb7dd70c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 14:37:00.565815 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:37:00.565772    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsccx\" (UniqueName: \"kubernetes.io/projected/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-kube-api-access-jsccx\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 14:37:00.565815 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:37:00.565811    2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-registry-tls\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 14:37:00.566016 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:37:00.565826    2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-image-registry-private-configuration\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 14:37:00.566016 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:37:00.565840    2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-installation-pull-secrets\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 14:37:00.566016 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:37:00.565851    2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-ca-trust-extracted\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 14:37:00.566016 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:37:00.565862    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-trusted-ca\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 14:37:00.566016 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:37:00.565872    2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-bound-sa-token\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 14:37:00.566016 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:37:00.565882    2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c-registry-certificates\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 14:37:01.151721 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:37:01.151686    2563 generic.go:356] "Generic (PLEG): container finished" podID="51a472c7-19c4-4c5c-9813-ebf6cb7dd70c" containerID="5596ef7131279e5949c22653301cfd753bdc4d6eb2b6979b956955c10f058a63" exitCode=0
Jul 29 14:37:01.152162 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:37:01.151733    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r"
Jul 29 14:37:01.152162 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:37:01.151753    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r" event={"ID":"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c","Type":"ContainerDied","Data":"5596ef7131279e5949c22653301cfd753bdc4d6eb2b6979b956955c10f058a63"}
Jul 29 14:37:01.152162 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:37:01.151795    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d8957c4d5-4pb5r" event={"ID":"51a472c7-19c4-4c5c-9813-ebf6cb7dd70c","Type":"ContainerDied","Data":"542b73f077e2436bfacdcdce42fd835f3006e16c154fc3406f082186dabbcef1"}
Jul 29 14:37:01.152162 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:37:01.151815    2563 scope.go:117] "RemoveContainer" containerID="5596ef7131279e5949c22653301cfd753bdc4d6eb2b6979b956955c10f058a63"
Jul 29 14:37:01.160549 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:37:01.160533    2563 scope.go:117] "RemoveContainer" containerID="5596ef7131279e5949c22653301cfd753bdc4d6eb2b6979b956955c10f058a63"
Jul 29 14:37:01.160800 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:37:01.160774    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5596ef7131279e5949c22653301cfd753bdc4d6eb2b6979b956955c10f058a63\": container with ID starting with 5596ef7131279e5949c22653301cfd753bdc4d6eb2b6979b956955c10f058a63 not found: ID does not exist" containerID="5596ef7131279e5949c22653301cfd753bdc4d6eb2b6979b956955c10f058a63"
Jul 29 14:37:01.160850 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:37:01.160810    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5596ef7131279e5949c22653301cfd753bdc4d6eb2b6979b956955c10f058a63"} err="failed to get container status \"5596ef7131279e5949c22653301cfd753bdc4d6eb2b6979b956955c10f058a63\": rpc error: code = NotFound desc = could not find container \"5596ef7131279e5949c22653301cfd753bdc4d6eb2b6979b956955c10f058a63\": container with ID starting with 5596ef7131279e5949c22653301cfd753bdc4d6eb2b6979b956955c10f058a63 not found: ID does not exist"
Jul 29 14:37:01.171420 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:37:01.171397    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d8957c4d5-4pb5r"]
Jul 29 14:37:01.175127 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:37:01.175108    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5d8957c4d5-4pb5r"]
Jul 29 14:37:02.431622 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:37:02.431583    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51a472c7-19c4-4c5c-9813-ebf6cb7dd70c" path="/var/lib/kubelet/pods/51a472c7-19c4-4c5c-9813-ebf6cb7dd70c/volumes"
Jul 29 14:37:05.169072 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:37:05.169037    2563 generic.go:356] "Generic (PLEG): container finished" podID="15611a65-9454-4396-9b2f-fc9599672fc4" containerID="b9e4bb84e6bf82d686e336e7ce65cbf1fd2b036cd0e44f032b5c14161aec98dc" exitCode=0
Jul 29 14:37:05.169494 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:37:05.169105    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-nh8pd" event={"ID":"15611a65-9454-4396-9b2f-fc9599672fc4","Type":"ContainerDied","Data":"b9e4bb84e6bf82d686e336e7ce65cbf1fd2b036cd0e44f032b5c14161aec98dc"}
Jul 29 14:37:05.169494 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:37:05.169462    2563 scope.go:117] "RemoveContainer" containerID="b9e4bb84e6bf82d686e336e7ce65cbf1fd2b036cd0e44f032b5c14161aec98dc"
Jul 29 14:37:06.176109 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:37:06.176057    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-nh8pd" event={"ID":"15611a65-9454-4396-9b2f-fc9599672fc4","Type":"ContainerStarted","Data":"e178bbc1a7f16b61d6b62f1609329f46cddf1c6d3dafceaadc89b0150598db58"}
Jul 29 14:39:58.330460 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:39:58.330433    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xnm95_23d79588-937d-4c6d-8760-a081024f608e/console-operator/1.log"
Jul 29 14:39:58.336999 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:39:58.336973    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xnm95_23d79588-937d-4c6d-8760-a081024f608e/console-operator/1.log"
Jul 29 14:39:58.339439 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:39:58.339415    2563 kubelet.go:1648] "Image garbage collection succeeded"
Jul 29 14:41:22.058715 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:22.058679    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-65f9f6cd8c-m78x8"]
Jul 29 14:41:22.059207 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:22.058992    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51a472c7-19c4-4c5c-9813-ebf6cb7dd70c" containerName="registry"
Jul 29 14:41:22.059207 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:22.059003    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="51a472c7-19c4-4c5c-9813-ebf6cb7dd70c" containerName="registry"
Jul 29 14:41:22.059207 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:22.059082    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="51a472c7-19c4-4c5c-9813-ebf6cb7dd70c" containerName="registry"
Jul 29 14:41:22.061698 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:22.061683    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-65f9f6cd8c-m78x8"
Jul 29 14:41:22.063998 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:22.063972    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 29 14:41:22.063998 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:22.063991    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 29 14:41:22.064218 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:22.064039    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 29 14:41:22.064218 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:22.064199    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-zq7td\""
Jul 29 14:41:22.064343 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:22.064328    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 29 14:41:22.084224 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:22.084195    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-65f9f6cd8c-m78x8"]
Jul 29 14:41:22.098538 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:22.098510    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be465c39-a202-4767-8e93-907564622462-tmp\") pod \"opendatahub-operator-controller-manager-65f9f6cd8c-m78x8\" (UID: \"be465c39-a202-4767-8e93-907564622462\") " pod="opendatahub/opendatahub-operator-controller-manager-65f9f6cd8c-m78x8"
Jul 29 14:41:22.098682 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:22.098583    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/be465c39-a202-4767-8e93-907564622462-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65f9f6cd8c-m78x8\" (UID: \"be465c39-a202-4767-8e93-907564622462\") " pod="opendatahub/opendatahub-operator-controller-manager-65f9f6cd8c-m78x8"
Jul 29 14:41:22.098682 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:22.098615    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w27qj\" (UniqueName: \"kubernetes.io/projected/be465c39-a202-4767-8e93-907564622462-kube-api-access-w27qj\") pod \"opendatahub-operator-controller-manager-65f9f6cd8c-m78x8\" (UID: \"be465c39-a202-4767-8e93-907564622462\") " pod="opendatahub/opendatahub-operator-controller-manager-65f9f6cd8c-m78x8"
Jul 29 14:41:22.098682 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:22.098644    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/be465c39-a202-4767-8e93-907564622462-manifests\") pod \"opendatahub-operator-controller-manager-65f9f6cd8c-m78x8\" (UID: \"be465c39-a202-4767-8e93-907564622462\") " pod="opendatahub/opendatahub-operator-controller-manager-65f9f6cd8c-m78x8"
Jul 29 14:41:22.098682 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:22.098667    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/be465c39-a202-4767-8e93-907564622462-webhook-cert\") pod \"opendatahub-operator-controller-manager-65f9f6cd8c-m78x8\" (UID: \"be465c39-a202-4767-8e93-907564622462\") " pod="opendatahub/opendatahub-operator-controller-manager-65f9f6cd8c-m78x8"
Jul 29 14:41:22.199798 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:22.199761    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be465c39-a202-4767-8e93-907564622462-tmp\") pod \"opendatahub-operator-controller-manager-65f9f6cd8c-m78x8\" (UID: \"be465c39-a202-4767-8e93-907564622462\") " pod="opendatahub/opendatahub-operator-controller-manager-65f9f6cd8c-m78x8"
Jul 29 14:41:22.199979 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:22.199860    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/be465c39-a202-4767-8e93-907564622462-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65f9f6cd8c-m78x8\" (UID: \"be465c39-a202-4767-8e93-907564622462\") " pod="opendatahub/opendatahub-operator-controller-manager-65f9f6cd8c-m78x8"
Jul 29 14:41:22.199979 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:22.199895    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w27qj\" (UniqueName: \"kubernetes.io/projected/be465c39-a202-4767-8e93-907564622462-kube-api-access-w27qj\") pod \"opendatahub-operator-controller-manager-65f9f6cd8c-m78x8\" (UID: \"be465c39-a202-4767-8e93-907564622462\") " pod="opendatahub/opendatahub-operator-controller-manager-65f9f6cd8c-m78x8"
Jul 29 14:41:22.199979 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:22.199928    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/be465c39-a202-4767-8e93-907564622462-manifests\") pod \"opendatahub-operator-controller-manager-65f9f6cd8c-m78x8\" (UID: \"be465c39-a202-4767-8e93-907564622462\") " pod="opendatahub/opendatahub-operator-controller-manager-65f9f6cd8c-m78x8"
Jul 29 14:41:22.199979 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:22.199952    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/be465c39-a202-4767-8e93-907564622462-webhook-cert\") pod \"opendatahub-operator-controller-manager-65f9f6cd8c-m78x8\" (UID: \"be465c39-a202-4767-8e93-907564622462\") " pod="opendatahub/opendatahub-operator-controller-manager-65f9f6cd8c-m78x8"
Jul 29 14:41:22.200369 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:22.200340    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/be465c39-a202-4767-8e93-907564622462-manifests\") pod \"opendatahub-operator-controller-manager-65f9f6cd8c-m78x8\" (UID: \"be465c39-a202-4767-8e93-907564622462\") " pod="opendatahub/opendatahub-operator-controller-manager-65f9f6cd8c-m78x8"
Jul 29 14:41:22.202303 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:22.202285    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be465c39-a202-4767-8e93-907564622462-tmp\") pod \"opendatahub-operator-controller-manager-65f9f6cd8c-m78x8\" (UID: \"be465c39-a202-4767-8e93-907564622462\") " pod="opendatahub/opendatahub-operator-controller-manager-65f9f6cd8c-m78x8"
Jul 29 14:41:22.202448 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:22.202427    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/be465c39-a202-4767-8e93-907564622462-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65f9f6cd8c-m78x8\" (UID: \"be465c39-a202-4767-8e93-907564622462\") " pod="opendatahub/opendatahub-operator-controller-manager-65f9f6cd8c-m78x8"
Jul 29 14:41:22.202517 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:22.202502    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/be465c39-a202-4767-8e93-907564622462-webhook-cert\") pod \"opendatahub-operator-controller-manager-65f9f6cd8c-m78x8\" (UID: \"be465c39-a202-4767-8e93-907564622462\") " pod="opendatahub/opendatahub-operator-controller-manager-65f9f6cd8c-m78x8"
Jul 29 14:41:22.208157 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:22.208115    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w27qj\" (UniqueName: \"kubernetes.io/projected/be465c39-a202-4767-8e93-907564622462-kube-api-access-w27qj\") pod \"opendatahub-operator-controller-manager-65f9f6cd8c-m78x8\" (UID: \"be465c39-a202-4767-8e93-907564622462\") " pod="opendatahub/opendatahub-operator-controller-manager-65f9f6cd8c-m78x8"
Jul 29 14:41:22.372068 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:22.371980    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-65f9f6cd8c-m78x8"
Jul 29 14:41:22.502262 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:22.502233    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-65f9f6cd8c-m78x8"]
Jul 29 14:41:22.505391 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:41:22.505364    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe465c39_a202_4767_8e93_907564622462.slice/crio-aaf3292c8bbaeebe8d9ce5c79d984c58097fd7534b2b53f8055460c747b947b1 WatchSource:0}: Error finding container aaf3292c8bbaeebe8d9ce5c79d984c58097fd7534b2b53f8055460c747b947b1: Status 404 returned error can't find the container with id aaf3292c8bbaeebe8d9ce5c79d984c58097fd7534b2b53f8055460c747b947b1
Jul 29 14:41:22.507123 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:22.507104    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 14:41:23.128958 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:23.128913    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-65f9f6cd8c-m78x8" event={"ID":"be465c39-a202-4767-8e93-907564622462","Type":"ContainerStarted","Data":"aaf3292c8bbaeebe8d9ce5c79d984c58097fd7534b2b53f8055460c747b947b1"}
Jul 29 14:41:26.147576 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:26.147534    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-65f9f6cd8c-m78x8" event={"ID":"be465c39-a202-4767-8e93-907564622462","Type":"ContainerStarted","Data":"6b5a161ec5e4b7b9f8d0fed009e6025244d75833362e0a5d860be18401e36d43"}
Jul 29 14:41:27.153544 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:27.153501    2563 generic.go:356] "Generic (PLEG): container finished" podID="be465c39-a202-4767-8e93-907564622462" containerID="6b5a161ec5e4b7b9f8d0fed009e6025244d75833362e0a5d860be18401e36d43" exitCode=0
Jul 29 14:41:27.153923 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:27.153557    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-65f9f6cd8c-m78x8" event={"ID":"be465c39-a202-4767-8e93-907564622462","Type":"ContainerDied","Data":"6b5a161ec5e4b7b9f8d0fed009e6025244d75833362e0a5d860be18401e36d43"}
Jul 29 14:41:28.159512 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:28.159475    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-65f9f6cd8c-m78x8" event={"ID":"be465c39-a202-4767-8e93-907564622462","Type":"ContainerStarted","Data":"c0f124ec0d0c1d4525dc86ac348614b5612f42c74662d66611bbf4133320fd58"}
Jul 29 14:41:28.159901 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:28.159579    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-65f9f6cd8c-m78x8"
Jul 29 14:41:28.187458 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:28.187402    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-65f9f6cd8c-m78x8" podStartSLOduration=1.686589603 podStartE2EDuration="6.187383982s" totalImagesPullingTime="3.406214206s" totalInitContainerRuntime="1.094580173s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:41:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:41:28.185544604 +0000 UTC m=+390.234002537" watchObservedRunningTime="2026-07-29 14:41:28.187383982 +0000 UTC m=+390.235841920"
Jul 29 14:41:39.166250 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:39.166169    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-65f9f6cd8c-m78x8"
Jul 29 14:41:49.907475 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:49.907439    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-856778fb54-t2kfl"]
Jul 29 14:41:49.920176 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:49.920126    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-856778fb54-t2kfl"]
Jul 29 14:41:49.920364 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:49.920214    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-856778fb54-t2kfl"
Jul 29 14:41:49.922757 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:49.922408    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 29 14:41:49.922757 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:49.922578    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 29 14:41:49.922757 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:49.922637    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 29 14:41:49.923250 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:49.923229    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 29 14:41:49.923250 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:49.923225    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-7cqkr\""
Jul 29 14:41:49.923418 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:49.923338    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 29 14:41:50.039250 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:50.039203    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f06b59e-f9ff-4761-b638-442843e96949-cert\") pod \"lws-controller-manager-856778fb54-t2kfl\" (UID: \"2f06b59e-f9ff-4761-b638-442843e96949\") " pod="openshift-lws-operator/lws-controller-manager-856778fb54-t2kfl"
Jul 29 14:41:50.039421 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:50.039292    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2f06b59e-f9ff-4761-b638-442843e96949-manager-config\") pod \"lws-controller-manager-856778fb54-t2kfl\" (UID: \"2f06b59e-f9ff-4761-b638-442843e96949\") " pod="openshift-lws-operator/lws-controller-manager-856778fb54-t2kfl"
Jul 29 14:41:50.039421 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:50.039333    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pwng\" (UniqueName: \"kubernetes.io/projected/2f06b59e-f9ff-4761-b638-442843e96949-kube-api-access-8pwng\") pod \"lws-controller-manager-856778fb54-t2kfl\" (UID: \"2f06b59e-f9ff-4761-b638-442843e96949\") " pod="openshift-lws-operator/lws-controller-manager-856778fb54-t2kfl"
Jul 29 14:41:50.039421 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:50.039358    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2f06b59e-f9ff-4761-b638-442843e96949-metrics-cert\") pod \"lws-controller-manager-856778fb54-t2kfl\" (UID: \"2f06b59e-f9ff-4761-b638-442843e96949\") " pod="openshift-lws-operator/lws-controller-manager-856778fb54-t2kfl"
Jul 29 14:41:50.140554 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:50.140513    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f06b59e-f9ff-4761-b638-442843e96949-cert\") pod \"lws-controller-manager-856778fb54-t2kfl\" (UID: \"2f06b59e-f9ff-4761-b638-442843e96949\") " pod="openshift-lws-operator/lws-controller-manager-856778fb54-t2kfl"
Jul 29 14:41:50.140719 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:50.140578    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2f06b59e-f9ff-4761-b638-442843e96949-manager-config\") pod \"lws-controller-manager-856778fb54-t2kfl\" (UID: \"2f06b59e-f9ff-4761-b638-442843e96949\") " pod="openshift-lws-operator/lws-controller-manager-856778fb54-t2kfl"
Jul 29 14:41:50.140719 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:50.140614    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pwng\" (UniqueName: \"kubernetes.io/projected/2f06b59e-f9ff-4761-b638-442843e96949-kube-api-access-8pwng\") pod \"lws-controller-manager-856778fb54-t2kfl\" (UID: \"2f06b59e-f9ff-4761-b638-442843e96949\") " pod="openshift-lws-operator/lws-controller-manager-856778fb54-t2kfl"
Jul 29 14:41:50.140719 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:50.140683    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2f06b59e-f9ff-4761-b638-442843e96949-metrics-cert\") pod \"lws-controller-manager-856778fb54-t2kfl\" (UID: \"2f06b59e-f9ff-4761-b638-442843e96949\") " pod="openshift-lws-operator/lws-controller-manager-856778fb54-t2kfl"
Jul 29 14:41:50.141359 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:50.141336    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2f06b59e-f9ff-4761-b638-442843e96949-manager-config\") pod \"lws-controller-manager-856778fb54-t2kfl\" (UID: \"2f06b59e-f9ff-4761-b638-442843e96949\") " pod="openshift-lws-operator/lws-controller-manager-856778fb54-t2kfl"
Jul 29 14:41:50.143177 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:50.143156    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f06b59e-f9ff-4761-b638-442843e96949-cert\") pod \"lws-controller-manager-856778fb54-t2kfl\" (UID: \"2f06b59e-f9ff-4761-b638-442843e96949\") " pod="openshift-lws-operator/lws-controller-manager-856778fb54-t2kfl"
Jul 29 14:41:50.143272 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:50.143206    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2f06b59e-f9ff-4761-b638-442843e96949-metrics-cert\") pod \"lws-controller-manager-856778fb54-t2kfl\" (UID: \"2f06b59e-f9ff-4761-b638-442843e96949\") " pod="openshift-lws-operator/lws-controller-manager-856778fb54-t2kfl"
Jul 29 14:41:50.148809 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:50.148784    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pwng\" (UniqueName: \"kubernetes.io/projected/2f06b59e-f9ff-4761-b638-442843e96949-kube-api-access-8pwng\") pod \"lws-controller-manager-856778fb54-t2kfl\" (UID: \"2f06b59e-f9ff-4761-b638-442843e96949\") " pod="openshift-lws-operator/lws-controller-manager-856778fb54-t2kfl"
Jul 29 14:41:50.230680 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:50.230647    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-856778fb54-t2kfl"
Jul 29 14:41:50.378778 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:50.378654    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-856778fb54-t2kfl"]
Jul 29 14:41:51.260417 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:51.260380    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-856778fb54-t2kfl" event={"ID":"2f06b59e-f9ff-4761-b638-442843e96949","Type":"ContainerStarted","Data":"b906e25c566210f25cd88a18b3834404fb59d52d6350a267351f3c9351eea186"}
Jul 29 14:41:53.275014 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:53.274975    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-856778fb54-t2kfl" event={"ID":"2f06b59e-f9ff-4761-b638-442843e96949","Type":"ContainerStarted","Data":"f643353540f5100d5ce6347d41a8a2cb667317cc1d28c899dbd7ccbae76c8e35"}
Jul 29 14:41:53.275802 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:53.275780    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-856778fb54-t2kfl"
Jul 29 14:41:53.374996 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:41:53.374931    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-856778fb54-t2kfl" podStartSLOduration=1.8113922759999999 podStartE2EDuration="4.374910671s" totalImagesPullingTime="2.563518395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:41:53.361028246 +0000 UTC m=+415.409486186" watchObservedRunningTime="2026-07-29 14:41:53.374910671 +0000 UTC m=+415.423368610"
Jul 29 14:42:05.286805 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:05.286774    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-856778fb54-t2kfl"
Jul 29 14:42:42.610227 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:42.610192    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9vbf5"]
Jul 29 14:42:42.615117 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:42.615096    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-9vbf5"
Jul 29 14:42:42.617252 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:42.617219    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 29 14:42:42.617928 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:42.617908    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 29 14:42:42.617928 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:42.617918    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-qnl4h\""
Jul 29 14:42:42.622579 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:42.622553    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9vbf5"]
Jul 29 14:42:42.780447 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:42.780397    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x4r6\" (UniqueName: \"kubernetes.io/projected/a6735af3-d5d5-4682-82b1-a60324c756af-kube-api-access-5x4r6\") pod \"kuadrant-operator-catalog-9vbf5\" (UID: \"a6735af3-d5d5-4682-82b1-a60324c756af\") " pod="kuadrant-system/kuadrant-operator-catalog-9vbf5"
Jul 29 14:42:42.881962 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:42.881861    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x4r6\" (UniqueName: \"kubernetes.io/projected/a6735af3-d5d5-4682-82b1-a60324c756af-kube-api-access-5x4r6\") pod \"kuadrant-operator-catalog-9vbf5\" (UID: \"a6735af3-d5d5-4682-82b1-a60324c756af\") " pod="kuadrant-system/kuadrant-operator-catalog-9vbf5"
Jul 29 14:42:42.890053 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:42.890023    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x4r6\" (UniqueName: \"kubernetes.io/projected/a6735af3-d5d5-4682-82b1-a60324c756af-kube-api-access-5x4r6\") pod \"kuadrant-operator-catalog-9vbf5\" (UID: \"a6735af3-d5d5-4682-82b1-a60324c756af\") " pod="kuadrant-system/kuadrant-operator-catalog-9vbf5"
Jul 29 14:42:42.926085 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:42.926041    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-9vbf5"
Jul 29 14:42:42.981652 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:42.981612    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9vbf5"]
Jul 29 14:42:43.054397 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:43.054282    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9vbf5"]
Jul 29 14:42:43.057034 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:42:43.057008    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6735af3_d5d5_4682_82b1_a60324c756af.slice/crio-b5b5a66a3ad5299840de598b400ed0a91cdc5ec68884cdb7da008cb37ef500ae WatchSource:0}: Error finding container b5b5a66a3ad5299840de598b400ed0a91cdc5ec68884cdb7da008cb37ef500ae: Status 404 returned error can't find the container with id b5b5a66a3ad5299840de598b400ed0a91cdc5ec68884cdb7da008cb37ef500ae
Jul 29 14:42:43.190441 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:43.190353    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-t8vp6"]
Jul 29 14:42:43.193549 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:43.193533    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-t8vp6"
Jul 29 14:42:43.200712 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:43.200687    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-t8vp6"]
Jul 29 14:42:43.286421 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:43.286382    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjtw7\" (UniqueName: \"kubernetes.io/projected/80ed5604-3b78-4658-8263-63d431ed86d5-kube-api-access-cjtw7\") pod \"kuadrant-operator-catalog-t8vp6\" (UID: \"80ed5604-3b78-4658-8263-63d431ed86d5\") " pod="kuadrant-system/kuadrant-operator-catalog-t8vp6"
Jul 29 14:42:43.387054 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:43.387020    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjtw7\" (UniqueName: \"kubernetes.io/projected/80ed5604-3b78-4658-8263-63d431ed86d5-kube-api-access-cjtw7\") pod \"kuadrant-operator-catalog-t8vp6\" (UID: \"80ed5604-3b78-4658-8263-63d431ed86d5\") " pod="kuadrant-system/kuadrant-operator-catalog-t8vp6"
Jul 29 14:42:43.395389 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:43.395357    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjtw7\" (UniqueName: \"kubernetes.io/projected/80ed5604-3b78-4658-8263-63d431ed86d5-kube-api-access-cjtw7\") pod \"kuadrant-operator-catalog-t8vp6\" (UID: \"80ed5604-3b78-4658-8263-63d431ed86d5\") " pod="kuadrant-system/kuadrant-operator-catalog-t8vp6"
Jul 29 14:42:43.478890 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:43.478849    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-9vbf5" event={"ID":"a6735af3-d5d5-4682-82b1-a60324c756af","Type":"ContainerStarted","Data":"b5b5a66a3ad5299840de598b400ed0a91cdc5ec68884cdb7da008cb37ef500ae"}
Jul 29 14:42:43.504288 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:43.504243    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-t8vp6"
Jul 29 14:42:43.640522 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:43.640489    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-t8vp6"]
Jul 29 14:42:43.643754 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:42:43.643724    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80ed5604_3b78_4658_8263_63d431ed86d5.slice/crio-116a007d3759fd8f46c290e25cb5cde71556435b4810ca697d67d05ed04c954f WatchSource:0}: Error finding container 116a007d3759fd8f46c290e25cb5cde71556435b4810ca697d67d05ed04c954f: Status 404 returned error can't find the container with id 116a007d3759fd8f46c290e25cb5cde71556435b4810ca697d67d05ed04c954f
Jul 29 14:42:44.487084 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:44.487048    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-t8vp6" event={"ID":"80ed5604-3b78-4658-8263-63d431ed86d5","Type":"ContainerStarted","Data":"116a007d3759fd8f46c290e25cb5cde71556435b4810ca697d67d05ed04c954f"}
Jul 29 14:42:45.494830 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:45.494780    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-t8vp6" event={"ID":"80ed5604-3b78-4658-8263-63d431ed86d5","Type":"ContainerStarted","Data":"5b669f58231fc7f6ab2570e994f56f5fa5d683a970c5f75715caa11a92719363"}
Jul 29 14:42:45.496687 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:45.496664    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-9vbf5" event={"ID":"a6735af3-d5d5-4682-82b1-a60324c756af","Type":"ContainerStarted","Data":"dccf308ea673a4a59187bd4cb26c6ef32e2cb8defae8d182bcc6bee7dc92c34b"}
Jul 29 14:42:45.496841 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:45.496817    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-9vbf5" podUID="a6735af3-d5d5-4682-82b1-a60324c756af" containerName="registry-server" containerID="cri-o://dccf308ea673a4a59187bd4cb26c6ef32e2cb8defae8d182bcc6bee7dc92c34b" gracePeriod=2
Jul 29 14:42:45.508893 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:45.508840    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-t8vp6" podStartSLOduration=0.767917371 podStartE2EDuration="2.50882225s" totalImagesPullingTime="1.740904879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:42:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:42:45.508765263 +0000 UTC m=+467.557223203" watchObservedRunningTime="2026-07-29 14:42:45.50882225 +0000 UTC m=+467.557280190"
Jul 29 14:42:45.523782 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:45.523729    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-9vbf5" podStartSLOduration=1.198928495 podStartE2EDuration="3.523713501s" totalImagesPullingTime="2.324785006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:42:45.522127784 +0000 UTC m=+467.570585724" watchObservedRunningTime="2026-07-29 14:42:45.523713501 +0000 UTC m=+467.572171441"
Jul 29 14:42:45.857447 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:45.857422    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-9vbf5"
Jul 29 14:42:46.011706 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:46.011557    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5x4r6\" (UniqueName: \"kubernetes.io/projected/a6735af3-d5d5-4682-82b1-a60324c756af-kube-api-access-5x4r6\") pod \"a6735af3-d5d5-4682-82b1-a60324c756af\" (UID: \"a6735af3-d5d5-4682-82b1-a60324c756af\") "
Jul 29 14:42:46.013719 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:46.013696    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6735af3-d5d5-4682-82b1-a60324c756af-kube-api-access-5x4r6" (OuterVolumeSpecName: "kube-api-access-5x4r6") pod "a6735af3-d5d5-4682-82b1-a60324c756af" (UID: "a6735af3-d5d5-4682-82b1-a60324c756af"). InnerVolumeSpecName "kube-api-access-5x4r6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 14:42:46.112950 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:46.112902    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x4r6\" (UniqueName: \"kubernetes.io/projected/a6735af3-d5d5-4682-82b1-a60324c756af-kube-api-access-5x4r6\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 14:42:46.501998 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:46.501960    2563 generic.go:356] "Generic (PLEG): container finished" podID="a6735af3-d5d5-4682-82b1-a60324c756af" containerID="dccf308ea673a4a59187bd4cb26c6ef32e2cb8defae8d182bcc6bee7dc92c34b" exitCode=0
Jul 29 14:42:46.502497 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:46.502007    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-9vbf5"
Jul 29 14:42:46.502497 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:46.502047    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-9vbf5" event={"ID":"a6735af3-d5d5-4682-82b1-a60324c756af","Type":"ContainerDied","Data":"dccf308ea673a4a59187bd4cb26c6ef32e2cb8defae8d182bcc6bee7dc92c34b"}
Jul 29 14:42:46.502497 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:46.502104    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-9vbf5" event={"ID":"a6735af3-d5d5-4682-82b1-a60324c756af","Type":"ContainerDied","Data":"b5b5a66a3ad5299840de598b400ed0a91cdc5ec68884cdb7da008cb37ef500ae"}
Jul 29 14:42:46.502497 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:46.502150    2563 scope.go:117] "RemoveContainer" containerID="dccf308ea673a4a59187bd4cb26c6ef32e2cb8defae8d182bcc6bee7dc92c34b"
Jul 29 14:42:46.511005 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:46.510983    2563 scope.go:117] "RemoveContainer" containerID="dccf308ea673a4a59187bd4cb26c6ef32e2cb8defae8d182bcc6bee7dc92c34b"
Jul 29 14:42:46.511302 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:42:46.511285    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dccf308ea673a4a59187bd4cb26c6ef32e2cb8defae8d182bcc6bee7dc92c34b\": container with ID starting with dccf308ea673a4a59187bd4cb26c6ef32e2cb8defae8d182bcc6bee7dc92c34b not found: ID does not exist" containerID="dccf308ea673a4a59187bd4cb26c6ef32e2cb8defae8d182bcc6bee7dc92c34b"
Jul 29 14:42:46.511352 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:46.511311    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dccf308ea673a4a59187bd4cb26c6ef32e2cb8defae8d182bcc6bee7dc92c34b"} err="failed to get container status \"dccf308ea673a4a59187bd4cb26c6ef32e2cb8defae8d182bcc6bee7dc92c34b\": rpc error: code = NotFound desc = could not find container \"dccf308ea673a4a59187bd4cb26c6ef32e2cb8defae8d182bcc6bee7dc92c34b\": container with ID starting with dccf308ea673a4a59187bd4cb26c6ef32e2cb8defae8d182bcc6bee7dc92c34b not found: ID does not exist"
Jul 29 14:42:46.516362 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:46.516338    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9vbf5"]
Jul 29 14:42:46.520059 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:46.520038    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9vbf5"]
Jul 29 14:42:48.431557 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:48.431526    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6735af3-d5d5-4682-82b1-a60324c756af" path="/var/lib/kubelet/pods/a6735af3-d5d5-4682-82b1-a60324c756af/volumes"
Jul 29 14:42:53.504750 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:53.504692    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-t8vp6"
Jul 29 14:42:53.505226 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:53.504775    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-t8vp6"
Jul 29 14:42:53.526889 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:53.526857    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-t8vp6"
Jul 29 14:42:53.554217 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:42:53.554190    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-t8vp6"
Jul 29 14:43:15.150063 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:15.150024    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jvnlq"]
Jul 29 14:43:15.150545 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:15.150349    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6735af3-d5d5-4682-82b1-a60324c756af" containerName="registry-server"
Jul 29 14:43:15.150545 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:15.150361    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6735af3-d5d5-4682-82b1-a60324c756af" containerName="registry-server"
Jul 29 14:43:15.150545 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:15.150435    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6735af3-d5d5-4682-82b1-a60324c756af" containerName="registry-server"
Jul 29 14:43:15.154606 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:15.154587    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jvnlq"
Jul 29 14:43:15.156832 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:15.156808    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-qg2fs\""
Jul 29 14:43:15.164544 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:15.164520    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jvnlq"]
Jul 29 14:43:15.244731 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:15.244695    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st55m\" (UniqueName: \"kubernetes.io/projected/63d6d159-f44a-4f3f-a209-4eaa4757ceb9-kube-api-access-st55m\") pod \"limitador-operator-controller-manager-859d7b78bd-jvnlq\" (UID: \"63d6d159-f44a-4f3f-a209-4eaa4757ceb9\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jvnlq"
Jul 29 14:43:15.345323 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:15.345288    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st55m\" (UniqueName: \"kubernetes.io/projected/63d6d159-f44a-4f3f-a209-4eaa4757ceb9-kube-api-access-st55m\") pod \"limitador-operator-controller-manager-859d7b78bd-jvnlq\" (UID: \"63d6d159-f44a-4f3f-a209-4eaa4757ceb9\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jvnlq"
Jul 29 14:43:15.357080 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:15.357052    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-st55m\" (UniqueName: \"kubernetes.io/projected/63d6d159-f44a-4f3f-a209-4eaa4757ceb9-kube-api-access-st55m\") pod \"limitador-operator-controller-manager-859d7b78bd-jvnlq\" (UID: \"63d6d159-f44a-4f3f-a209-4eaa4757ceb9\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jvnlq"
Jul 29 14:43:15.464895 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:15.464868    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jvnlq"
Jul 29 14:43:15.598626 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:15.598527    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jvnlq"]
Jul 29 14:43:15.601534 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:43:15.601504    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d6d159_f44a_4f3f_a209_4eaa4757ceb9.slice/crio-b6dc560f89c4ed6236da1ec951d6492d1db61a1e7f0545c315548cef3cb97a91 WatchSource:0}: Error finding container b6dc560f89c4ed6236da1ec951d6492d1db61a1e7f0545c315548cef3cb97a91: Status 404 returned error can't find the container with id b6dc560f89c4ed6236da1ec951d6492d1db61a1e7f0545c315548cef3cb97a91
Jul 29 14:43:15.627053 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:15.627013    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jvnlq" event={"ID":"63d6d159-f44a-4f3f-a209-4eaa4757ceb9","Type":"ContainerStarted","Data":"b6dc560f89c4ed6236da1ec951d6492d1db61a1e7f0545c315548cef3cb97a91"}
Jul 29 14:43:17.637355 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:17.637316    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jvnlq" event={"ID":"63d6d159-f44a-4f3f-a209-4eaa4757ceb9","Type":"ContainerStarted","Data":"4155aa613bd7ae729e2df3d0abfbe5b61e296a28bca578ac4d7fff4e865e687c"}
Jul 29 14:43:17.637769 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:17.637490    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jvnlq"
Jul 29 14:43:17.671995 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:17.671945    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jvnlq" podStartSLOduration=0.798685561 podStartE2EDuration="2.671930853s" totalImagesPullingTime="1.873245292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:43:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:43:17.670593238 +0000 UTC m=+499.719051178" watchObservedRunningTime="2026-07-29 14:43:17.671930853 +0000 UTC m=+499.720388813"
Jul 29 14:43:22.678194 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:22.678153    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-bf7t2"]
Jul 29 14:43:22.681378 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:22.681361    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bf7t2"
Jul 29 14:43:22.684417 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:22.684394    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 29 14:43:22.684549 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:22.684531    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-zz2b2\""
Jul 29 14:43:22.693419 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:22.693395    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-bf7t2"]
Jul 29 14:43:22.812238 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:22.812198    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg9jt\" (UniqueName: \"kubernetes.io/projected/93d6e8f8-836e-4538-9326-c81fd86c1fe0-kube-api-access-fg9jt\") pod \"dns-operator-controller-manager-864dd78c4c-bf7t2\" (UID: \"93d6e8f8-836e-4538-9326-c81fd86c1fe0\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bf7t2"
Jul 29 14:43:22.913029 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:22.912990    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg9jt\" (UniqueName: \"kubernetes.io/projected/93d6e8f8-836e-4538-9326-c81fd86c1fe0-kube-api-access-fg9jt\") pod \"dns-operator-controller-manager-864dd78c4c-bf7t2\" (UID: \"93d6e8f8-836e-4538-9326-c81fd86c1fe0\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bf7t2"
Jul 29 14:43:22.921051 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:22.921018    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg9jt\" (UniqueName: \"kubernetes.io/projected/93d6e8f8-836e-4538-9326-c81fd86c1fe0-kube-api-access-fg9jt\") pod \"dns-operator-controller-manager-864dd78c4c-bf7t2\" (UID: \"93d6e8f8-836e-4538-9326-c81fd86c1fe0\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bf7t2"
Jul 29 14:43:22.991022 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:22.990993    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bf7t2"
Jul 29 14:43:23.120297 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:23.120264    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-bf7t2"]
Jul 29 14:43:23.123576 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:43:23.123550    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93d6e8f8_836e_4538_9326_c81fd86c1fe0.slice/crio-cb82ea0c6353c7fe5ba258aac5c6c297d7f4d1f4fb34582479088294a5f0b4e6 WatchSource:0}: Error finding container cb82ea0c6353c7fe5ba258aac5c6c297d7f4d1f4fb34582479088294a5f0b4e6: Status 404 returned error can't find the container with id cb82ea0c6353c7fe5ba258aac5c6c297d7f4d1f4fb34582479088294a5f0b4e6
Jul 29 14:43:23.663663 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:23.663622    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bf7t2" event={"ID":"93d6e8f8-836e-4538-9326-c81fd86c1fe0","Type":"ContainerStarted","Data":"cb82ea0c6353c7fe5ba258aac5c6c297d7f4d1f4fb34582479088294a5f0b4e6"}
Jul 29 14:43:25.675109 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:25.675020    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bf7t2" event={"ID":"93d6e8f8-836e-4538-9326-c81fd86c1fe0","Type":"ContainerStarted","Data":"045ed8a73f4b948b3b6997c3b6c3afc561ad90006c778822f9118fdd901afa9d"}
Jul 29 14:43:25.675528 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:25.675121    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bf7t2"
Jul 29 14:43:25.692308 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:25.692237    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bf7t2" podStartSLOduration=1.449803454 podStartE2EDuration="3.692221043s" totalImagesPullingTime="2.242417589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:43:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:43:25.690738133 +0000 UTC m=+507.739196072" watchObservedRunningTime="2026-07-29 14:43:25.692221043 +0000 UTC m=+507.740678982"
Jul 29 14:43:28.643729 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:28.643701    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jvnlq"
Jul 29 14:43:36.681893 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:36.681860    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bf7t2"
Jul 29 14:43:38.890859 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:38.890830    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jvnlq"]
Jul 29 14:43:38.891252 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:38.891026    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jvnlq" podUID="63d6d159-f44a-4f3f-a209-4eaa4757ceb9" containerName="manager" containerID="cri-o://4155aa613bd7ae729e2df3d0abfbe5b61e296a28bca578ac4d7fff4e865e687c" gracePeriod=2
Jul 29 14:43:38.910268 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:38.910243    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jvnlq"]
Jul 29 14:43:38.918023 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:43:38.917985    2563 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"63d6d159-f44a-4f3f-a209-4eaa4757ceb9\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-jvnlq\": pods \"limitador-operator-controller-manager-859d7b78bd-jvnlq\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jvnlq"
Jul 29 14:43:38.920156 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:38.920119    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8s4rc"]
Jul 29 14:43:38.920447 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:38.920429    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63d6d159-f44a-4f3f-a209-4eaa4757ceb9" containerName="manager"
Jul 29 14:43:38.920447 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:38.920443    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="63d6d159-f44a-4f3f-a209-4eaa4757ceb9" containerName="manager"
Jul 29 14:43:38.920579 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:38.920534    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="63d6d159-f44a-4f3f-a209-4eaa4757ceb9" containerName="manager"
Jul 29 14:43:38.923745 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:38.923727    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8s4rc"
Jul 29 14:43:38.936411 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:38.936387    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8s4rc"]
Jul 29 14:43:38.956732 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:43:38.956698    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jvnlq\" is forbidden: User \"system:node:ip-10-0-136-196.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-196.ec2.internal' and this object" podUID="63d6d159-f44a-4f3f-a209-4eaa4757ceb9" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jvnlq"
Jul 29 14:43:39.051049 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:39.050944    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2qlf\" (UniqueName: \"kubernetes.io/projected/7445057f-7eb6-4532-ae7b-e9422ce7cb7f-kube-api-access-l2qlf\") pod \"limitador-operator-controller-manager-859d7b78bd-8s4rc\" (UID: \"7445057f-7eb6-4532-ae7b-e9422ce7cb7f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8s4rc"
Jul 29 14:43:39.130588 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:39.130560    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jvnlq"
Jul 29 14:43:39.132510 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:43:39.132481    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jvnlq\" is forbidden: User \"system:node:ip-10-0-136-196.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-196.ec2.internal' and this object" podUID="63d6d159-f44a-4f3f-a209-4eaa4757ceb9" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jvnlq"
Jul 29 14:43:39.151927 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:39.151899    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2qlf\" (UniqueName: \"kubernetes.io/projected/7445057f-7eb6-4532-ae7b-e9422ce7cb7f-kube-api-access-l2qlf\") pod \"limitador-operator-controller-manager-859d7b78bd-8s4rc\" (UID: \"7445057f-7eb6-4532-ae7b-e9422ce7cb7f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8s4rc"
Jul 29 14:43:39.164960 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:39.164926    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2qlf\" (UniqueName: \"kubernetes.io/projected/7445057f-7eb6-4532-ae7b-e9422ce7cb7f-kube-api-access-l2qlf\") pod \"limitador-operator-controller-manager-859d7b78bd-8s4rc\" (UID: \"7445057f-7eb6-4532-ae7b-e9422ce7cb7f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8s4rc"
Jul 29 14:43:39.253125 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:39.253069    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-st55m\" (UniqueName: \"kubernetes.io/projected/63d6d159-f44a-4f3f-a209-4eaa4757ceb9-kube-api-access-st55m\") pod \"63d6d159-f44a-4f3f-a209-4eaa4757ceb9\" (UID: \"63d6d159-f44a-4f3f-a209-4eaa4757ceb9\") "
Jul 29 14:43:39.255104 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:39.255081    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d6d159-f44a-4f3f-a209-4eaa4757ceb9-kube-api-access-st55m" (OuterVolumeSpecName: "kube-api-access-st55m") pod "63d6d159-f44a-4f3f-a209-4eaa4757ceb9" (UID: "63d6d159-f44a-4f3f-a209-4eaa4757ceb9"). InnerVolumeSpecName "kube-api-access-st55m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 14:43:39.272397 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:39.272370    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8s4rc"
Jul 29 14:43:39.354717 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:39.354674    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st55m\" (UniqueName: \"kubernetes.io/projected/63d6d159-f44a-4f3f-a209-4eaa4757ceb9-kube-api-access-st55m\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 14:43:39.407358 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:39.407321    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8s4rc"]
Jul 29 14:43:39.411658 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:43:39.411634    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7445057f_7eb6_4532_ae7b_e9422ce7cb7f.slice/crio-5c8773b996ccb2bd42b77cd1e870af5d9929fa4a89c04a6fa60b5c35d39030e1 WatchSource:0}: Error finding container 5c8773b996ccb2bd42b77cd1e870af5d9929fa4a89c04a6fa60b5c35d39030e1: Status 404 returned error can't find the container with id 5c8773b996ccb2bd42b77cd1e870af5d9929fa4a89c04a6fa60b5c35d39030e1
Jul 29 14:43:39.737726 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:39.737689    2563 generic.go:356] "Generic (PLEG): container finished" podID="63d6d159-f44a-4f3f-a209-4eaa4757ceb9" containerID="4155aa613bd7ae729e2df3d0abfbe5b61e296a28bca578ac4d7fff4e865e687c" exitCode=0
Jul 29 14:43:39.737907 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:39.737737    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jvnlq"
Jul 29 14:43:39.737907 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:39.737797    2563 scope.go:117] "RemoveContainer" containerID="4155aa613bd7ae729e2df3d0abfbe5b61e296a28bca578ac4d7fff4e865e687c"
Jul 29 14:43:39.739540 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:39.739516    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8s4rc" event={"ID":"7445057f-7eb6-4532-ae7b-e9422ce7cb7f","Type":"ContainerStarted","Data":"a626f83c8e7e7c597b95628b04f07600614a97005503c5234bec37e52715d905"}
Jul 29 14:43:39.739646 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:39.739563    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8s4rc" event={"ID":"7445057f-7eb6-4532-ae7b-e9422ce7cb7f","Type":"ContainerStarted","Data":"5c8773b996ccb2bd42b77cd1e870af5d9929fa4a89c04a6fa60b5c35d39030e1"}
Jul 29 14:43:39.739692 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:39.739643    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8s4rc"
Jul 29 14:43:39.740067 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:43:39.740037    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jvnlq\" is forbidden: User \"system:node:ip-10-0-136-196.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-196.ec2.internal' and this object" podUID="63d6d159-f44a-4f3f-a209-4eaa4757ceb9" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jvnlq"
Jul 29 14:43:39.747039 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:39.747020    2563 scope.go:117] "RemoveContainer" containerID="4155aa613bd7ae729e2df3d0abfbe5b61e296a28bca578ac4d7fff4e865e687c"
Jul 29 14:43:39.747306 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:43:39.747286    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4155aa613bd7ae729e2df3d0abfbe5b61e296a28bca578ac4d7fff4e865e687c\": container with ID starting with 4155aa613bd7ae729e2df3d0abfbe5b61e296a28bca578ac4d7fff4e865e687c not found: ID does not exist" containerID="4155aa613bd7ae729e2df3d0abfbe5b61e296a28bca578ac4d7fff4e865e687c"
Jul 29 14:43:39.747369 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:39.747315    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4155aa613bd7ae729e2df3d0abfbe5b61e296a28bca578ac4d7fff4e865e687c"} err="failed to get container status \"4155aa613bd7ae729e2df3d0abfbe5b61e296a28bca578ac4d7fff4e865e687c\": rpc error: code = NotFound desc = could not find container \"4155aa613bd7ae729e2df3d0abfbe5b61e296a28bca578ac4d7fff4e865e687c\": container with ID starting with 4155aa613bd7ae729e2df3d0abfbe5b61e296a28bca578ac4d7fff4e865e687c not found: ID does not exist"
Jul 29 14:43:39.768891 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:39.768847    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8s4rc" podStartSLOduration=1.7688330030000001 podStartE2EDuration="1.768833003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:43:39.767537333 +0000 UTC m=+521.815995273" watchObservedRunningTime="2026-07-29 14:43:39.768833003 +0000 UTC m=+521.817291015"
Jul 29 14:43:39.769977 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:43:39.769949    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jvnlq\" is forbidden: User \"system:node:ip-10-0-136-196.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-196.ec2.internal' and this object" podUID="63d6d159-f44a-4f3f-a209-4eaa4757ceb9" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jvnlq"
Jul 29 14:43:39.771512 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:43:39.771491    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jvnlq\" is forbidden: User \"system:node:ip-10-0-136-196.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-196.ec2.internal' and this object" podUID="63d6d159-f44a-4f3f-a209-4eaa4757ceb9" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jvnlq"
Jul 29 14:43:40.432000 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:40.431962    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63d6d159-f44a-4f3f-a209-4eaa4757ceb9" path="/var/lib/kubelet/pods/63d6d159-f44a-4f3f-a209-4eaa4757ceb9/volumes"
Jul 29 14:43:50.747767 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:43:50.747684    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8s4rc"
Jul 29 14:44:22.214638 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:22.214601    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-9bhrp"]
Jul 29 14:44:22.217936 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:22.217913    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-9bhrp"
Jul 29 14:44:22.219978 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:22.219956    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 29 14:44:22.220085 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:22.220055    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bvthn\""
Jul 29 14:44:22.227584 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:22.227559    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-9bhrp"]
Jul 29 14:44:22.297074 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:22.297023    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gns4q\" (UniqueName: \"kubernetes.io/projected/a76f0192-b8b3-4d1a-bb46-9fc458cc2ab3-kube-api-access-gns4q\") pod \"limitador-limitador-64c94fd64f-9bhrp\" (UID: \"a76f0192-b8b3-4d1a-bb46-9fc458cc2ab3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-9bhrp"
Jul 29 14:44:22.297321 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:22.297123    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a76f0192-b8b3-4d1a-bb46-9fc458cc2ab3-config-file\") pod \"limitador-limitador-64c94fd64f-9bhrp\" (UID: \"a76f0192-b8b3-4d1a-bb46-9fc458cc2ab3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-9bhrp"
Jul 29 14:44:22.313395 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:22.313356    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-9bhrp"]
Jul 29 14:44:22.398493 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:22.398452    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gns4q\" (UniqueName: \"kubernetes.io/projected/a76f0192-b8b3-4d1a-bb46-9fc458cc2ab3-kube-api-access-gns4q\") pod \"limitador-limitador-64c94fd64f-9bhrp\" (UID: \"a76f0192-b8b3-4d1a-bb46-9fc458cc2ab3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-9bhrp"
Jul 29 14:44:22.398669 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:22.398525    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a76f0192-b8b3-4d1a-bb46-9fc458cc2ab3-config-file\") pod \"limitador-limitador-64c94fd64f-9bhrp\" (UID: \"a76f0192-b8b3-4d1a-bb46-9fc458cc2ab3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-9bhrp"
Jul 29 14:44:22.399182 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:22.399126    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a76f0192-b8b3-4d1a-bb46-9fc458cc2ab3-config-file\") pod \"limitador-limitador-64c94fd64f-9bhrp\" (UID: \"a76f0192-b8b3-4d1a-bb46-9fc458cc2ab3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-9bhrp"
Jul 29 14:44:22.406335 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:22.406309    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gns4q\" (UniqueName: \"kubernetes.io/projected/a76f0192-b8b3-4d1a-bb46-9fc458cc2ab3-kube-api-access-gns4q\") pod \"limitador-limitador-64c94fd64f-9bhrp\" (UID: \"a76f0192-b8b3-4d1a-bb46-9fc458cc2ab3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-9bhrp"
Jul 29 14:44:22.528696 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:22.528597    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-9bhrp"
Jul 29 14:44:22.663922 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:22.663897    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-9bhrp"]
Jul 29 14:44:22.666611 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:44:22.666583    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda76f0192_b8b3_4d1a_bb46_9fc458cc2ab3.slice/crio-a8766ab7bb66293c61fe489124aa6efc435f758af763962e995e84fec69167e5 WatchSource:0}: Error finding container a8766ab7bb66293c61fe489124aa6efc435f758af763962e995e84fec69167e5: Status 404 returned error can't find the container with id a8766ab7bb66293c61fe489124aa6efc435f758af763962e995e84fec69167e5
Jul 29 14:44:22.930195 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:22.930083    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-9bhrp" event={"ID":"a76f0192-b8b3-4d1a-bb46-9fc458cc2ab3","Type":"ContainerStarted","Data":"a8766ab7bb66293c61fe489124aa6efc435f758af763962e995e84fec69167e5"}
Jul 29 14:44:23.029543 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:23.029512    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-9kmkg"]
Jul 29 14:44:23.033810 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:23.033791    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-9kmkg"
Jul 29 14:44:23.035894 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:23.035869    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-6zqnh\""
Jul 29 14:44:23.041799 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:23.041773    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-9kmkg"]
Jul 29 14:44:23.104829 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:23.104799    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qpff\" (UniqueName: \"kubernetes.io/projected/90c8b218-0a3e-41a9-bde0-40ad32caac04-kube-api-access-6qpff\") pod \"authorino-58cd955675-9kmkg\" (UID: \"90c8b218-0a3e-41a9-bde0-40ad32caac04\") " pod="kuadrant-system/authorino-58cd955675-9kmkg"
Jul 29 14:44:23.205952 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:23.205917    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qpff\" (UniqueName: \"kubernetes.io/projected/90c8b218-0a3e-41a9-bde0-40ad32caac04-kube-api-access-6qpff\") pod \"authorino-58cd955675-9kmkg\" (UID: \"90c8b218-0a3e-41a9-bde0-40ad32caac04\") " pod="kuadrant-system/authorino-58cd955675-9kmkg"
Jul 29 14:44:23.213578 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:23.213553    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qpff\" (UniqueName: \"kubernetes.io/projected/90c8b218-0a3e-41a9-bde0-40ad32caac04-kube-api-access-6qpff\") pod \"authorino-58cd955675-9kmkg\" (UID: \"90c8b218-0a3e-41a9-bde0-40ad32caac04\") " pod="kuadrant-system/authorino-58cd955675-9kmkg"
Jul 29 14:44:23.344191 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:23.344150    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-9kmkg"
Jul 29 14:44:23.471212 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:23.470976    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-9kmkg"]
Jul 29 14:44:23.473779 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:44:23.473745    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90c8b218_0a3e_41a9_bde0_40ad32caac04.slice/crio-b9935b2cdc3127cbd281b912db75a491cbd23cd4ce19958105b07a772917b5e3 WatchSource:0}: Error finding container b9935b2cdc3127cbd281b912db75a491cbd23cd4ce19958105b07a772917b5e3: Status 404 returned error can't find the container with id b9935b2cdc3127cbd281b912db75a491cbd23cd4ce19958105b07a772917b5e3
Jul 29 14:44:23.935861 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:23.935782    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-9kmkg" event={"ID":"90c8b218-0a3e-41a9-bde0-40ad32caac04","Type":"ContainerStarted","Data":"b9935b2cdc3127cbd281b912db75a491cbd23cd4ce19958105b07a772917b5e3"}
Jul 29 14:44:28.569016 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:28.568925    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-9kmkg"]
Jul 29 14:44:28.963160 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:28.963092    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-9kmkg" event={"ID":"90c8b218-0a3e-41a9-bde0-40ad32caac04","Type":"ContainerStarted","Data":"383d387a0c64493efd1ee40364f76b2d6902d32baa732b57264965413de8e6b0"}
Jul 29 14:44:28.964472 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:28.964451    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-9bhrp" event={"ID":"a76f0192-b8b3-4d1a-bb46-9fc458cc2ab3","Type":"ContainerStarted","Data":"3ba4640230d8ba75ba0064ac1980c838bee70f9cdb55941530602afac2f67280"}
Jul 29 14:44:28.964617 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:28.964606    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-9bhrp"
Jul 29 14:44:28.979673 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:28.979627    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-9kmkg" podStartSLOduration=1.204237878 podStartE2EDuration="5.979615583s" totalImagesPullingTime="4.775377705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:44:28.977702374 +0000 UTC m=+571.026160314" watchObservedRunningTime="2026-07-29 14:44:28.979615583 +0000 UTC m=+571.028073531"
Jul 29 14:44:28.994504 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:28.994462    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-9bhrp" podStartSLOduration=1.413605932 podStartE2EDuration="6.994449764s" totalImagesPullingTime="5.580843832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:44:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:44:28.993797382 +0000 UTC m=+571.042255322" watchObservedRunningTime="2026-07-29 14:44:28.994449764 +0000 UTC m=+571.042907696"
Jul 29 14:44:29.968565 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:29.968518    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-9kmkg" podUID="90c8b218-0a3e-41a9-bde0-40ad32caac04" containerName="authorino" containerID="cri-o://383d387a0c64493efd1ee40364f76b2d6902d32baa732b57264965413de8e6b0" gracePeriod=30
Jul 29 14:44:30.210747 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:30.210719    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-9kmkg"
Jul 29 14:44:30.265744 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:30.265652    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qpff\" (UniqueName: \"kubernetes.io/projected/90c8b218-0a3e-41a9-bde0-40ad32caac04-kube-api-access-6qpff\") pod \"90c8b218-0a3e-41a9-bde0-40ad32caac04\" (UID: \"90c8b218-0a3e-41a9-bde0-40ad32caac04\") "
Jul 29 14:44:30.267657 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:30.267622    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90c8b218-0a3e-41a9-bde0-40ad32caac04-kube-api-access-6qpff" (OuterVolumeSpecName: "kube-api-access-6qpff") pod "90c8b218-0a3e-41a9-bde0-40ad32caac04" (UID: "90c8b218-0a3e-41a9-bde0-40ad32caac04"). InnerVolumeSpecName "kube-api-access-6qpff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 14:44:30.367063 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:30.367025    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qpff\" (UniqueName: \"kubernetes.io/projected/90c8b218-0a3e-41a9-bde0-40ad32caac04-kube-api-access-6qpff\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 14:44:30.974873 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:30.974837    2563 generic.go:356] "Generic (PLEG): container finished" podID="90c8b218-0a3e-41a9-bde0-40ad32caac04" containerID="383d387a0c64493efd1ee40364f76b2d6902d32baa732b57264965413de8e6b0" exitCode=0
Jul 29 14:44:30.975345 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:30.974882    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-9kmkg" event={"ID":"90c8b218-0a3e-41a9-bde0-40ad32caac04","Type":"ContainerDied","Data":"383d387a0c64493efd1ee40364f76b2d6902d32baa732b57264965413de8e6b0"}
Jul 29 14:44:30.975345 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:30.974889    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-9kmkg"
Jul 29 14:44:30.975345 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:30.974920    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-9kmkg" event={"ID":"90c8b218-0a3e-41a9-bde0-40ad32caac04","Type":"ContainerDied","Data":"b9935b2cdc3127cbd281b912db75a491cbd23cd4ce19958105b07a772917b5e3"}
Jul 29 14:44:30.975345 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:30.974945    2563 scope.go:117] "RemoveContainer" containerID="383d387a0c64493efd1ee40364f76b2d6902d32baa732b57264965413de8e6b0"
Jul 29 14:44:30.983314 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:30.983296    2563 scope.go:117] "RemoveContainer" containerID="383d387a0c64493efd1ee40364f76b2d6902d32baa732b57264965413de8e6b0"
Jul 29 14:44:30.983544 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:44:30.983525    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"383d387a0c64493efd1ee40364f76b2d6902d32baa732b57264965413de8e6b0\": container with ID starting with 383d387a0c64493efd1ee40364f76b2d6902d32baa732b57264965413de8e6b0 not found: ID does not exist" containerID="383d387a0c64493efd1ee40364f76b2d6902d32baa732b57264965413de8e6b0"
Jul 29 14:44:30.983600 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:30.983552    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"383d387a0c64493efd1ee40364f76b2d6902d32baa732b57264965413de8e6b0"} err="failed to get container status \"383d387a0c64493efd1ee40364f76b2d6902d32baa732b57264965413de8e6b0\": rpc error: code = NotFound desc = could not find container \"383d387a0c64493efd1ee40364f76b2d6902d32baa732b57264965413de8e6b0\": container with ID starting with 383d387a0c64493efd1ee40364f76b2d6902d32baa732b57264965413de8e6b0 not found: ID does not exist"
Jul 29 14:44:30.988904 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:30.988883    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-9kmkg"]
Jul 29 14:44:30.991677 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:30.991658    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-9kmkg"]
Jul 29 14:44:32.431844 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:32.431804    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90c8b218-0a3e-41a9-bde0-40ad32caac04" path="/var/lib/kubelet/pods/90c8b218-0a3e-41a9-bde0-40ad32caac04/volumes"
Jul 29 14:44:39.544557 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:39.544522    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-9bhrp"]
Jul 29 14:44:39.545018 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:39.544716    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-9bhrp" podUID="a76f0192-b8b3-4d1a-bb46-9fc458cc2ab3" containerName="limitador" containerID="cri-o://3ba4640230d8ba75ba0064ac1980c838bee70f9cdb55941530602afac2f67280" gracePeriod=30
Jul 29 14:44:39.547100 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:39.547076    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-9bhrp"
Jul 29 14:44:40.015439 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:40.015407    2563 generic.go:356] "Generic (PLEG): container finished" podID="a76f0192-b8b3-4d1a-bb46-9fc458cc2ab3" containerID="3ba4640230d8ba75ba0064ac1980c838bee70f9cdb55941530602afac2f67280" exitCode=0
Jul 29 14:44:40.015595 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:40.015473    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-9bhrp" event={"ID":"a76f0192-b8b3-4d1a-bb46-9fc458cc2ab3","Type":"ContainerDied","Data":"3ba4640230d8ba75ba0064ac1980c838bee70f9cdb55941530602afac2f67280"}
Jul 29 14:44:40.492081 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:40.492055    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-9bhrp"
Jul 29 14:44:40.649935 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:40.649827    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a76f0192-b8b3-4d1a-bb46-9fc458cc2ab3-config-file\") pod \"a76f0192-b8b3-4d1a-bb46-9fc458cc2ab3\" (UID: \"a76f0192-b8b3-4d1a-bb46-9fc458cc2ab3\") "
Jul 29 14:44:40.649935 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:40.649925    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gns4q\" (UniqueName: \"kubernetes.io/projected/a76f0192-b8b3-4d1a-bb46-9fc458cc2ab3-kube-api-access-gns4q\") pod \"a76f0192-b8b3-4d1a-bb46-9fc458cc2ab3\" (UID: \"a76f0192-b8b3-4d1a-bb46-9fc458cc2ab3\") "
Jul 29 14:44:40.650392 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:40.650211    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a76f0192-b8b3-4d1a-bb46-9fc458cc2ab3-config-file" (OuterVolumeSpecName: "config-file") pod "a76f0192-b8b3-4d1a-bb46-9fc458cc2ab3" (UID: "a76f0192-b8b3-4d1a-bb46-9fc458cc2ab3"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 14:44:40.651970 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:40.651942    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a76f0192-b8b3-4d1a-bb46-9fc458cc2ab3-kube-api-access-gns4q" (OuterVolumeSpecName: "kube-api-access-gns4q") pod "a76f0192-b8b3-4d1a-bb46-9fc458cc2ab3" (UID: "a76f0192-b8b3-4d1a-bb46-9fc458cc2ab3"). InnerVolumeSpecName "kube-api-access-gns4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 14:44:40.750596 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:40.750562    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gns4q\" (UniqueName: \"kubernetes.io/projected/a76f0192-b8b3-4d1a-bb46-9fc458cc2ab3-kube-api-access-gns4q\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 14:44:40.750596 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:40.750593    2563 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a76f0192-b8b3-4d1a-bb46-9fc458cc2ab3-config-file\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 14:44:41.020923 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:41.020893    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-9bhrp"
Jul 29 14:44:41.021114 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:41.020900    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-9bhrp" event={"ID":"a76f0192-b8b3-4d1a-bb46-9fc458cc2ab3","Type":"ContainerDied","Data":"a8766ab7bb66293c61fe489124aa6efc435f758af763962e995e84fec69167e5"}
Jul 29 14:44:41.021114 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:41.021027    2563 scope.go:117] "RemoveContainer" containerID="3ba4640230d8ba75ba0064ac1980c838bee70f9cdb55941530602afac2f67280"
Jul 29 14:44:41.045533 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:41.045504    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-9bhrp"]
Jul 29 14:44:41.053721 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:41.053696    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-9bhrp"]
Jul 29 14:44:41.535156 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:41.535101    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-kq4j9"]
Jul 29 14:44:41.535519 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:41.535499    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a76f0192-b8b3-4d1a-bb46-9fc458cc2ab3" containerName="limitador"
Jul 29 14:44:41.535591 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:41.535522    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a76f0192-b8b3-4d1a-bb46-9fc458cc2ab3" containerName="limitador"
Jul 29 14:44:41.535591 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:41.535546    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90c8b218-0a3e-41a9-bde0-40ad32caac04" containerName="authorino"
Jul 29 14:44:41.535591 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:41.535555    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="90c8b218-0a3e-41a9-bde0-40ad32caac04" containerName="authorino"
Jul 29 14:44:41.535685 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:41.535627    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="90c8b218-0a3e-41a9-bde0-40ad32caac04" containerName="authorino"
Jul 29 14:44:41.535685 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:41.535645    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a76f0192-b8b3-4d1a-bb46-9fc458cc2ab3" containerName="limitador"
Jul 29 14:44:41.540152 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:41.540112    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kq4j9"
Jul 29 14:44:41.542215 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:41.542185    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 14:44:41.542346 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:41.542320    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 29 14:44:41.542439 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:41.542376    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-6t2d2\""
Jul 29 14:44:41.542504 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:41.542436    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 14:44:41.546482 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:41.546462    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-kq4j9"]
Jul 29 14:44:41.657310 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:41.657266    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c3cf1288-f7de-424e-b18c-f3e8c882d07c-data\") pod \"postgres-69fb4594cf-kq4j9\" (UID: \"c3cf1288-f7de-424e-b18c-f3e8c882d07c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kq4j9"
Jul 29 14:44:41.657684 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:41.657329    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk7x7\" (UniqueName: \"kubernetes.io/projected/c3cf1288-f7de-424e-b18c-f3e8c882d07c-kube-api-access-xk7x7\") pod \"postgres-69fb4594cf-kq4j9\" (UID: \"c3cf1288-f7de-424e-b18c-f3e8c882d07c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kq4j9"
Jul 29 14:44:41.758262 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:41.758221    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c3cf1288-f7de-424e-b18c-f3e8c882d07c-data\") pod \"postgres-69fb4594cf-kq4j9\" (UID: \"c3cf1288-f7de-424e-b18c-f3e8c882d07c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kq4j9"
Jul 29 14:44:41.758462 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:41.758271    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk7x7\" (UniqueName: \"kubernetes.io/projected/c3cf1288-f7de-424e-b18c-f3e8c882d07c-kube-api-access-xk7x7\") pod \"postgres-69fb4594cf-kq4j9\" (UID: \"c3cf1288-f7de-424e-b18c-f3e8c882d07c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kq4j9"
Jul 29 14:44:41.758611 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:41.758589    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c3cf1288-f7de-424e-b18c-f3e8c882d07c-data\") pod \"postgres-69fb4594cf-kq4j9\" (UID: \"c3cf1288-f7de-424e-b18c-f3e8c882d07c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kq4j9"
Jul 29 14:44:41.766659 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:41.766630    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk7x7\" (UniqueName: \"kubernetes.io/projected/c3cf1288-f7de-424e-b18c-f3e8c882d07c-kube-api-access-xk7x7\") pod \"postgres-69fb4594cf-kq4j9\" (UID: \"c3cf1288-f7de-424e-b18c-f3e8c882d07c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kq4j9"
Jul 29 14:44:41.851789 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:41.851710    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kq4j9"
Jul 29 14:44:41.982197 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:41.982168    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-kq4j9"]
Jul 29 14:44:41.984646 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:44:41.984622    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3cf1288_f7de_424e_b18c_f3e8c882d07c.slice/crio-75a5afce18d97b737aee0a87bbf34dcca13ace5da48890a3f00a6ee39672516f WatchSource:0}: Error finding container 75a5afce18d97b737aee0a87bbf34dcca13ace5da48890a3f00a6ee39672516f: Status 404 returned error can't find the container with id 75a5afce18d97b737aee0a87bbf34dcca13ace5da48890a3f00a6ee39672516f
Jul 29 14:44:42.030224 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:42.030189    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kq4j9" event={"ID":"c3cf1288-f7de-424e-b18c-f3e8c882d07c","Type":"ContainerStarted","Data":"75a5afce18d97b737aee0a87bbf34dcca13ace5da48890a3f00a6ee39672516f"}
Jul 29 14:44:42.431639 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:42.431604    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a76f0192-b8b3-4d1a-bb46-9fc458cc2ab3" path="/var/lib/kubelet/pods/a76f0192-b8b3-4d1a-bb46-9fc458cc2ab3/volumes"
Jul 29 14:44:47.059927 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:47.059849    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kq4j9" event={"ID":"c3cf1288-f7de-424e-b18c-f3e8c882d07c","Type":"ContainerStarted","Data":"4a7e63a7e2fceb13f0e040f64d8f50a0ac8a40110bf95a72f55c9c75b7c40595"}
Jul 29 14:44:47.059927 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:47.059918    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kq4j9"
Jul 29 14:44:47.076672 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:47.076613    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kq4j9" podStartSLOduration=1.2726553059999999 podStartE2EDuration="6.076592002s" totalImagesPullingTime="4.803936696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:44:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:44:47.074057712 +0000 UTC m=+589.122515652" watchObservedRunningTime="2026-07-29 14:44:47.076592002 +0000 UTC m=+589.125049941"
Jul 29 14:44:53.093105 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:53.093077    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kq4j9"
Jul 29 14:44:53.670590 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:53.670556    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7nc2h"]
Jul 29 14:44:53.672844 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:53.672828    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-7nc2h"
Jul 29 14:44:53.674873 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:53.674852    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-6zqnh\""
Jul 29 14:44:53.682001 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:53.681978    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7nc2h"]
Jul 29 14:44:53.764797 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:53.764762    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zff8x\" (UniqueName: \"kubernetes.io/projected/ffc8f758-12bf-475d-a450-94b6977a454c-kube-api-access-zff8x\") pod \"authorino-5d4b67c446-7nc2h\" (UID: \"ffc8f758-12bf-475d-a450-94b6977a454c\") " pod="kuadrant-system/authorino-5d4b67c446-7nc2h"
Jul 29 14:44:53.856556 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:53.856521    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7nc2h"]
Jul 29 14:44:53.856807 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:44:53.856783    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-zff8x], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-7nc2h" podUID="ffc8f758-12bf-475d-a450-94b6977a454c"
Jul 29 14:44:53.865929 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:53.865897    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zff8x\" (UniqueName: \"kubernetes.io/projected/ffc8f758-12bf-475d-a450-94b6977a454c-kube-api-access-zff8x\") pod \"authorino-5d4b67c446-7nc2h\" (UID: \"ffc8f758-12bf-475d-a450-94b6977a454c\") " pod="kuadrant-system/authorino-5d4b67c446-7nc2h"
Jul 29 14:44:53.877074 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:53.877042    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zff8x\" (UniqueName: \"kubernetes.io/projected/ffc8f758-12bf-475d-a450-94b6977a454c-kube-api-access-zff8x\") pod \"authorino-5d4b67c446-7nc2h\" (UID: \"ffc8f758-12bf-475d-a450-94b6977a454c\") " pod="kuadrant-system/authorino-5d4b67c446-7nc2h"
Jul 29 14:44:53.882401 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:53.882376    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-sdxq5"]
Jul 29 14:44:53.884814 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:53.884795    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-sdxq5"
Jul 29 14:44:53.886634 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:53.886615    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 29 14:44:53.893629 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:53.893606    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-sdxq5"]
Jul 29 14:44:53.930796 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:53.930710    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-sdxq5"]
Jul 29 14:44:53.930967 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:44:53.930946    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-4pf2j tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-4pf2j tls-cert]: context canceled" pod="kuadrant-system/authorino-76877994-sdxq5" podUID="068cf485-7f4f-4b92-b718-147c97e7eb3d"
Jul 29 14:44:53.960846 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:53.960822    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-66b68bbbfd-vlhgs"]
Jul 29 14:44:53.963689 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:53.963671    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66b68bbbfd-vlhgs"
Jul 29 14:44:53.971385 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:53.971364    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66b68bbbfd-vlhgs"]
Jul 29 14:44:54.067939 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:54.067900    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/068cf485-7f4f-4b92-b718-147c97e7eb3d-tls-cert\") pod \"authorino-76877994-sdxq5\" (UID: \"068cf485-7f4f-4b92-b718-147c97e7eb3d\") " pod="kuadrant-system/authorino-76877994-sdxq5"
Jul 29 14:44:54.067939 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:54.067937    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pf2j\" (UniqueName: \"kubernetes.io/projected/068cf485-7f4f-4b92-b718-147c97e7eb3d-kube-api-access-4pf2j\") pod \"authorino-76877994-sdxq5\" (UID: \"068cf485-7f4f-4b92-b718-147c97e7eb3d\") " pod="kuadrant-system/authorino-76877994-sdxq5"
Jul 29 14:44:54.068196 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:54.068016    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9e97e830-b9a1-467a-95b4-08ddd21c28c7-tls-cert\") pod \"authorino-66b68bbbfd-vlhgs\" (UID: \"9e97e830-b9a1-467a-95b4-08ddd21c28c7\") " pod="kuadrant-system/authorino-66b68bbbfd-vlhgs"
Jul 29 14:44:54.068196 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:54.068094    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82d6v\" (UniqueName: \"kubernetes.io/projected/9e97e830-b9a1-467a-95b4-08ddd21c28c7-kube-api-access-82d6v\") pod \"authorino-66b68bbbfd-vlhgs\" (UID: \"9e97e830-b9a1-467a-95b4-08ddd21c28c7\") " pod="kuadrant-system/authorino-66b68bbbfd-vlhgs"
Jul 29 14:44:54.090792 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:54.090766    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-sdxq5"
Jul 29 14:44:54.090933 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:54.090765    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-7nc2h"
Jul 29 14:44:54.096793 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:54.096771    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-sdxq5"
Jul 29 14:44:54.099542 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:54.099523    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-7nc2h"
Jul 29 14:44:54.169505 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:54.169465    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9e97e830-b9a1-467a-95b4-08ddd21c28c7-tls-cert\") pod \"authorino-66b68bbbfd-vlhgs\" (UID: \"9e97e830-b9a1-467a-95b4-08ddd21c28c7\") " pod="kuadrant-system/authorino-66b68bbbfd-vlhgs"
Jul 29 14:44:54.169683 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:54.169512    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82d6v\" (UniqueName: \"kubernetes.io/projected/9e97e830-b9a1-467a-95b4-08ddd21c28c7-kube-api-access-82d6v\") pod \"authorino-66b68bbbfd-vlhgs\" (UID: \"9e97e830-b9a1-467a-95b4-08ddd21c28c7\") " pod="kuadrant-system/authorino-66b68bbbfd-vlhgs"
Jul 29 14:44:54.169683 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:54.169668    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/068cf485-7f4f-4b92-b718-147c97e7eb3d-tls-cert\") pod \"authorino-76877994-sdxq5\" (UID: \"068cf485-7f4f-4b92-b718-147c97e7eb3d\") " pod="kuadrant-system/authorino-76877994-sdxq5"
Jul 29 14:44:54.169818 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:54.169703    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pf2j\" (UniqueName: \"kubernetes.io/projected/068cf485-7f4f-4b92-b718-147c97e7eb3d-kube-api-access-4pf2j\") pod \"authorino-76877994-sdxq5\" (UID: \"068cf485-7f4f-4b92-b718-147c97e7eb3d\") " pod="kuadrant-system/authorino-76877994-sdxq5"
Jul 29 14:44:54.172015 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:54.171984    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9e97e830-b9a1-467a-95b4-08ddd21c28c7-tls-cert\") pod \"authorino-66b68bbbfd-vlhgs\" (UID: \"9e97e830-b9a1-467a-95b4-08ddd21c28c7\") " pod="kuadrant-system/authorino-66b68bbbfd-vlhgs"
Jul 29 14:44:54.172160 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:54.172069    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/068cf485-7f4f-4b92-b718-147c97e7eb3d-tls-cert\") pod \"authorino-76877994-sdxq5\" (UID: \"068cf485-7f4f-4b92-b718-147c97e7eb3d\") " pod="kuadrant-system/authorino-76877994-sdxq5"
Jul 29 14:44:54.177687 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:54.177653    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82d6v\" (UniqueName: \"kubernetes.io/projected/9e97e830-b9a1-467a-95b4-08ddd21c28c7-kube-api-access-82d6v\") pod \"authorino-66b68bbbfd-vlhgs\" (UID: \"9e97e830-b9a1-467a-95b4-08ddd21c28c7\") " pod="kuadrant-system/authorino-66b68bbbfd-vlhgs"
Jul 29 14:44:54.177799 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:54.177725    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pf2j\" (UniqueName: \"kubernetes.io/projected/068cf485-7f4f-4b92-b718-147c97e7eb3d-kube-api-access-4pf2j\") pod \"authorino-76877994-sdxq5\" (UID: \"068cf485-7f4f-4b92-b718-147c97e7eb3d\") " pod="kuadrant-system/authorino-76877994-sdxq5"
Jul 29 14:44:54.270851 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:54.270816    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/068cf485-7f4f-4b92-b718-147c97e7eb3d-tls-cert\") pod \"068cf485-7f4f-4b92-b718-147c97e7eb3d\" (UID: \"068cf485-7f4f-4b92-b718-147c97e7eb3d\") "
Jul 29 14:44:54.271016 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:54.270910    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zff8x\" (UniqueName: \"kubernetes.io/projected/ffc8f758-12bf-475d-a450-94b6977a454c-kube-api-access-zff8x\") pod \"ffc8f758-12bf-475d-a450-94b6977a454c\" (UID: \"ffc8f758-12bf-475d-a450-94b6977a454c\") "
Jul 29 14:44:54.272994 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:54.272960    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/068cf485-7f4f-4b92-b718-147c97e7eb3d-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "068cf485-7f4f-4b92-b718-147c97e7eb3d" (UID: "068cf485-7f4f-4b92-b718-147c97e7eb3d"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 14:44:54.272994 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:54.272985    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66b68bbbfd-vlhgs"
Jul 29 14:44:54.273209 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:54.273010    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc8f758-12bf-475d-a450-94b6977a454c-kube-api-access-zff8x" (OuterVolumeSpecName: "kube-api-access-zff8x") pod "ffc8f758-12bf-475d-a450-94b6977a454c" (UID: "ffc8f758-12bf-475d-a450-94b6977a454c"). InnerVolumeSpecName "kube-api-access-zff8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 14:44:54.371977 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:54.371932    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pf2j\" (UniqueName: \"kubernetes.io/projected/068cf485-7f4f-4b92-b718-147c97e7eb3d-kube-api-access-4pf2j\") pod \"068cf485-7f4f-4b92-b718-147c97e7eb3d\" (UID: \"068cf485-7f4f-4b92-b718-147c97e7eb3d\") "
Jul 29 14:44:54.372284 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:54.372263    2563 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/068cf485-7f4f-4b92-b718-147c97e7eb3d-tls-cert\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 14:44:54.372339 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:54.372292    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zff8x\" (UniqueName: \"kubernetes.io/projected/ffc8f758-12bf-475d-a450-94b6977a454c-kube-api-access-zff8x\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 14:44:54.374004 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:54.373968    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/068cf485-7f4f-4b92-b718-147c97e7eb3d-kube-api-access-4pf2j" (OuterVolumeSpecName: "kube-api-access-4pf2j") pod "068cf485-7f4f-4b92-b718-147c97e7eb3d" (UID: "068cf485-7f4f-4b92-b718-147c97e7eb3d"). InnerVolumeSpecName "kube-api-access-4pf2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 14:44:54.398390 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:54.398250    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66b68bbbfd-vlhgs"]
Jul 29 14:44:54.400920 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:44:54.400894    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e97e830_b9a1_467a_95b4_08ddd21c28c7.slice/crio-ff618b97e1649ee7f64e422f25630b740b9bc2d55b831d707728fb6235ddde29 WatchSource:0}: Error finding container ff618b97e1649ee7f64e422f25630b740b9bc2d55b831d707728fb6235ddde29: Status 404 returned error can't find the container with id ff618b97e1649ee7f64e422f25630b740b9bc2d55b831d707728fb6235ddde29
Jul 29 14:44:54.473531 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:54.473500    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pf2j\" (UniqueName: \"kubernetes.io/projected/068cf485-7f4f-4b92-b718-147c97e7eb3d-kube-api-access-4pf2j\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 14:44:55.096600 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:55.096501    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66b68bbbfd-vlhgs" event={"ID":"9e97e830-b9a1-467a-95b4-08ddd21c28c7","Type":"ContainerStarted","Data":"4c9873a203662df73c80c9ea444dbc7c9ebfa4993b4fc5703f71224235ab1f8c"}
Jul 29 14:44:55.096600 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:55.096514    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-7nc2h"
Jul 29 14:44:55.096837 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:55.096523    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-sdxq5"
Jul 29 14:44:55.096837 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:55.096557    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66b68bbbfd-vlhgs" event={"ID":"9e97e830-b9a1-467a-95b4-08ddd21c28c7","Type":"ContainerStarted","Data":"ff618b97e1649ee7f64e422f25630b740b9bc2d55b831d707728fb6235ddde29"}
Jul 29 14:44:55.111912 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:55.111867    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-66b68bbbfd-vlhgs" podStartSLOduration=1.7652064520000001 podStartE2EDuration="2.111853003s" totalImagesPullingTime="346.646551ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:44:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:44:55.110620186 +0000 UTC m=+597.159078126" watchObservedRunningTime="2026-07-29 14:44:55.111853003 +0000 UTC m=+597.160310942"
Jul 29 14:44:55.133374 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:55.133344    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7nc2h"]
Jul 29 14:44:55.139690 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:55.139666    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7nc2h"]
Jul 29 14:44:55.164201 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:55.164166    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-sdxq5"]
Jul 29 14:44:55.171297 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:55.171272    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-sdxq5"]
Jul 29 14:44:56.434170 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:56.432591    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="068cf485-7f4f-4b92-b718-147c97e7eb3d" path="/var/lib/kubelet/pods/068cf485-7f4f-4b92-b718-147c97e7eb3d/volumes"
Jul 29 14:44:56.434170 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:56.432963    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffc8f758-12bf-475d-a450-94b6977a454c" path="/var/lib/kubelet/pods/ffc8f758-12bf-475d-a450-94b6977a454c/volumes"
Jul 29 14:44:58.358380 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:58.358355    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xnm95_23d79588-937d-4c6d-8760-a081024f608e/console-operator/1.log"
Jul 29 14:44:58.369721 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:58.369700    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xnm95_23d79588-937d-4c6d-8760-a081024f608e/console-operator/1.log"
Jul 29 14:44:59.964183 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:44:59.964148    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8bbd64c5-hpzxn"]
Jul 29 14:45:00.047391 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:00.047352    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8bbd64c5-hpzxn"]
Jul 29 14:45:00.047555 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:00.047491    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8bbd64c5-hpzxn"
Jul 29 14:45:00.049855 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:00.049830    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 29 14:45:00.049976 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:00.049859    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-57frt\""
Jul 29 14:45:00.049976 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:00.049833    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 29 14:45:00.116825 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:00.116795    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tnhv\" (UniqueName: \"kubernetes.io/projected/5b91a1c6-4e48-416b-8ccd-635f144f4168-kube-api-access-9tnhv\") pod \"maas-controller-8bbd64c5-hpzxn\" (UID: \"5b91a1c6-4e48-416b-8ccd-635f144f4168\") " pod="opendatahub/maas-controller-8bbd64c5-hpzxn"
Jul 29 14:45:00.116825 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:00.116829    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b91a1c6-4e48-416b-8ccd-635f144f4168-webhook-cert\") pod \"maas-controller-8bbd64c5-hpzxn\" (UID: \"5b91a1c6-4e48-416b-8ccd-635f144f4168\") " pod="opendatahub/maas-controller-8bbd64c5-hpzxn"
Jul 29 14:45:00.158323 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:00.158285    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-794c9d7b98-g46sn"]
Jul 29 14:45:00.185022 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:00.184990    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-794c9d7b98-g46sn"]
Jul 29 14:45:00.185178 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:00.185084    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-794c9d7b98-g46sn"
Jul 29 14:45:00.217857 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:00.217779    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzh48\" (UniqueName: \"kubernetes.io/projected/4cc52c77-cd0d-415c-8bce-c4132bfe66f2-kube-api-access-fzh48\") pod \"maas-controller-794c9d7b98-g46sn\" (UID: \"4cc52c77-cd0d-415c-8bce-c4132bfe66f2\") " pod="opendatahub/maas-controller-794c9d7b98-g46sn"
Jul 29 14:45:00.217857 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:00.217818    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4cc52c77-cd0d-415c-8bce-c4132bfe66f2-webhook-cert\") pod \"maas-controller-794c9d7b98-g46sn\" (UID: \"4cc52c77-cd0d-415c-8bce-c4132bfe66f2\") " pod="opendatahub/maas-controller-794c9d7b98-g46sn"
Jul 29 14:45:00.217857 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:00.217847    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tnhv\" (UniqueName: \"kubernetes.io/projected/5b91a1c6-4e48-416b-8ccd-635f144f4168-kube-api-access-9tnhv\") pod \"maas-controller-8bbd64c5-hpzxn\" (UID: \"5b91a1c6-4e48-416b-8ccd-635f144f4168\") " pod="opendatahub/maas-controller-8bbd64c5-hpzxn"
Jul 29 14:45:00.218049 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:00.217894    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b91a1c6-4e48-416b-8ccd-635f144f4168-webhook-cert\") pod \"maas-controller-8bbd64c5-hpzxn\" (UID: \"5b91a1c6-4e48-416b-8ccd-635f144f4168\") " pod="opendatahub/maas-controller-8bbd64c5-hpzxn"
Jul 29 14:45:00.220261 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:00.220240    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b91a1c6-4e48-416b-8ccd-635f144f4168-webhook-cert\") pod \"maas-controller-8bbd64c5-hpzxn\" (UID: \"5b91a1c6-4e48-416b-8ccd-635f144f4168\") " pod="opendatahub/maas-controller-8bbd64c5-hpzxn"
Jul 29 14:45:00.226394 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:00.226371    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tnhv\" (UniqueName: \"kubernetes.io/projected/5b91a1c6-4e48-416b-8ccd-635f144f4168-kube-api-access-9tnhv\") pod \"maas-controller-8bbd64c5-hpzxn\" (UID: \"5b91a1c6-4e48-416b-8ccd-635f144f4168\") " pod="opendatahub/maas-controller-8bbd64c5-hpzxn"
Jul 29 14:45:00.318731 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:00.318694    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4cc52c77-cd0d-415c-8bce-c4132bfe66f2-webhook-cert\") pod \"maas-controller-794c9d7b98-g46sn\" (UID: \"4cc52c77-cd0d-415c-8bce-c4132bfe66f2\") " pod="opendatahub/maas-controller-794c9d7b98-g46sn"
Jul 29 14:45:00.318911 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:00.318788    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzh48\" (UniqueName: \"kubernetes.io/projected/4cc52c77-cd0d-415c-8bce-c4132bfe66f2-kube-api-access-fzh48\") pod \"maas-controller-794c9d7b98-g46sn\" (UID: \"4cc52c77-cd0d-415c-8bce-c4132bfe66f2\") " pod="opendatahub/maas-controller-794c9d7b98-g46sn"
Jul 29 14:45:00.321126 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:00.321099    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4cc52c77-cd0d-415c-8bce-c4132bfe66f2-webhook-cert\") pod \"maas-controller-794c9d7b98-g46sn\" (UID: \"4cc52c77-cd0d-415c-8bce-c4132bfe66f2\") " pod="opendatahub/maas-controller-794c9d7b98-g46sn"
Jul 29 14:45:00.326957 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:00.326939    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzh48\" (UniqueName: \"kubernetes.io/projected/4cc52c77-cd0d-415c-8bce-c4132bfe66f2-kube-api-access-fzh48\") pod \"maas-controller-794c9d7b98-g46sn\" (UID: \"4cc52c77-cd0d-415c-8bce-c4132bfe66f2\") " pod="opendatahub/maas-controller-794c9d7b98-g46sn"
Jul 29 14:45:00.357508 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:00.357469    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8bbd64c5-hpzxn"
Jul 29 14:45:00.495442 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:00.495363    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-794c9d7b98-g46sn"
Jul 29 14:45:00.513704 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:00.513672    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8bbd64c5-hpzxn"]
Jul 29 14:45:00.517189 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:45:00.517043    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b91a1c6_4e48_416b_8ccd_635f144f4168.slice/crio-573f703785156eb5690d3141d619596d2772ffd383ffa4b1643ec3d3f0e35bec WatchSource:0}: Error finding container 573f703785156eb5690d3141d619596d2772ffd383ffa4b1643ec3d3f0e35bec: Status 404 returned error can't find the container with id 573f703785156eb5690d3141d619596d2772ffd383ffa4b1643ec3d3f0e35bec
Jul 29 14:45:00.642844 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:00.642818    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-794c9d7b98-g46sn"]
Jul 29 14:45:00.645254 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:45:00.645228    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cc52c77_cd0d_415c_8bce_c4132bfe66f2.slice/crio-ed404e68fb2c1bc81306eafae867debbfdc5e50e2d8def175a61a77ba81f5ba7 WatchSource:0}: Error finding container ed404e68fb2c1bc81306eafae867debbfdc5e50e2d8def175a61a77ba81f5ba7: Status 404 returned error can't find the container with id ed404e68fb2c1bc81306eafae867debbfdc5e50e2d8def175a61a77ba81f5ba7
Jul 29 14:45:01.126272 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:01.126240    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-794c9d7b98-g46sn" event={"ID":"4cc52c77-cd0d-415c-8bce-c4132bfe66f2","Type":"ContainerStarted","Data":"ed404e68fb2c1bc81306eafae867debbfdc5e50e2d8def175a61a77ba81f5ba7"}
Jul 29 14:45:01.127268 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:01.127243    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8bbd64c5-hpzxn" event={"ID":"5b91a1c6-4e48-416b-8ccd-635f144f4168","Type":"ContainerStarted","Data":"573f703785156eb5690d3141d619596d2772ffd383ffa4b1643ec3d3f0e35bec"}
Jul 29 14:45:04.149992 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:04.149948    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-794c9d7b98-g46sn" event={"ID":"4cc52c77-cd0d-415c-8bce-c4132bfe66f2","Type":"ContainerStarted","Data":"f1e6bc45a36b432660c84e15ecd314f97c6f067a47aa2e4dbb9f1884a6a50479"}
Jul 29 14:45:04.150444 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:04.150014    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-794c9d7b98-g46sn"
Jul 29 14:45:04.151582 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:04.151552    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8bbd64c5-hpzxn" event={"ID":"5b91a1c6-4e48-416b-8ccd-635f144f4168","Type":"ContainerStarted","Data":"14d5674f398aee1f91cdf2c44c7d8a0a8ca85d66c137360b68fe4f95ee9907a8"}
Jul 29 14:45:04.151690 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:04.151602    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8bbd64c5-hpzxn"
Jul 29 14:45:04.167363 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:04.167303    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-794c9d7b98-g46sn" podStartSLOduration=1.136500612 podStartE2EDuration="4.167284125s" totalImagesPullingTime="3.030783513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:45:04.166571834 +0000 UTC m=+606.215029775" watchObservedRunningTime="2026-07-29 14:45:04.167284125 +0000 UTC m=+606.215742065"
Jul 29 14:45:04.186195 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:04.186123    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-8bbd64c5-hpzxn" podStartSLOduration=2.022486822 podStartE2EDuration="5.186101582s" totalImagesPullingTime="3.16361476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:44:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:45:04.184369855 +0000 UTC m=+606.232827792" watchObservedRunningTime="2026-07-29 14:45:04.186101582 +0000 UTC m=+606.234559521"
Jul 29 14:45:15.160111 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:15.160078    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8bbd64c5-hpzxn"
Jul 29 14:45:15.160672 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:15.160164    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-794c9d7b98-g46sn"
Jul 29 14:45:15.245490 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:15.245454    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8bbd64c5-hpzxn"]
Jul 29 14:45:15.245741 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:15.245712    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-8bbd64c5-hpzxn" podUID="5b91a1c6-4e48-416b-8ccd-635f144f4168" containerName="manager" containerID="cri-o://14d5674f398aee1f91cdf2c44c7d8a0a8ca85d66c137360b68fe4f95ee9907a8" gracePeriod=30
Jul 29 14:45:15.495213 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:15.495188    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8bbd64c5-hpzxn"
Jul 29 14:45:15.547597 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:15.547569    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tnhv\" (UniqueName: \"kubernetes.io/projected/5b91a1c6-4e48-416b-8ccd-635f144f4168-kube-api-access-9tnhv\") pod \"5b91a1c6-4e48-416b-8ccd-635f144f4168\" (UID: \"5b91a1c6-4e48-416b-8ccd-635f144f4168\") "
Jul 29 14:45:15.547765 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:15.547606    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b91a1c6-4e48-416b-8ccd-635f144f4168-webhook-cert\") pod \"5b91a1c6-4e48-416b-8ccd-635f144f4168\" (UID: \"5b91a1c6-4e48-416b-8ccd-635f144f4168\") "
Jul 29 14:45:15.549769 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:15.549741    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b91a1c6-4e48-416b-8ccd-635f144f4168-kube-api-access-9tnhv" (OuterVolumeSpecName: "kube-api-access-9tnhv") pod "5b91a1c6-4e48-416b-8ccd-635f144f4168" (UID: "5b91a1c6-4e48-416b-8ccd-635f144f4168"). InnerVolumeSpecName "kube-api-access-9tnhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 14:45:15.549865 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:15.549779    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b91a1c6-4e48-416b-8ccd-635f144f4168-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "5b91a1c6-4e48-416b-8ccd-635f144f4168" (UID: "5b91a1c6-4e48-416b-8ccd-635f144f4168"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 14:45:15.649061 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:15.649028    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tnhv\" (UniqueName: \"kubernetes.io/projected/5b91a1c6-4e48-416b-8ccd-635f144f4168-kube-api-access-9tnhv\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 14:45:15.649061 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:15.649057    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b91a1c6-4e48-416b-8ccd-635f144f4168-webhook-cert\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 14:45:16.209996 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:16.209963    2563 generic.go:356] "Generic (PLEG): container finished" podID="5b91a1c6-4e48-416b-8ccd-635f144f4168" containerID="14d5674f398aee1f91cdf2c44c7d8a0a8ca85d66c137360b68fe4f95ee9907a8" exitCode=0
Jul 29 14:45:16.210389 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:16.210010    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8bbd64c5-hpzxn"
Jul 29 14:45:16.210389 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:16.210013    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8bbd64c5-hpzxn" event={"ID":"5b91a1c6-4e48-416b-8ccd-635f144f4168","Type":"ContainerDied","Data":"14d5674f398aee1f91cdf2c44c7d8a0a8ca85d66c137360b68fe4f95ee9907a8"}
Jul 29 14:45:16.210389 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:16.210070    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8bbd64c5-hpzxn" event={"ID":"5b91a1c6-4e48-416b-8ccd-635f144f4168","Type":"ContainerDied","Data":"573f703785156eb5690d3141d619596d2772ffd383ffa4b1643ec3d3f0e35bec"}
Jul 29 14:45:16.210389 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:16.210101    2563 scope.go:117] "RemoveContainer" containerID="14d5674f398aee1f91cdf2c44c7d8a0a8ca85d66c137360b68fe4f95ee9907a8"
Jul 29 14:45:16.218890 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:16.218874    2563 scope.go:117] "RemoveContainer" containerID="14d5674f398aee1f91cdf2c44c7d8a0a8ca85d66c137360b68fe4f95ee9907a8"
Jul 29 14:45:16.219204 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:45:16.219184    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14d5674f398aee1f91cdf2c44c7d8a0a8ca85d66c137360b68fe4f95ee9907a8\": container with ID starting with 14d5674f398aee1f91cdf2c44c7d8a0a8ca85d66c137360b68fe4f95ee9907a8 not found: ID does not exist" containerID="14d5674f398aee1f91cdf2c44c7d8a0a8ca85d66c137360b68fe4f95ee9907a8"
Jul 29 14:45:16.219258 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:16.219213    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14d5674f398aee1f91cdf2c44c7d8a0a8ca85d66c137360b68fe4f95ee9907a8"} err="failed to get container status \"14d5674f398aee1f91cdf2c44c7d8a0a8ca85d66c137360b68fe4f95ee9907a8\": rpc error: code = NotFound desc = could not find container \"14d5674f398aee1f91cdf2c44c7d8a0a8ca85d66c137360b68fe4f95ee9907a8\": container with ID starting with 14d5674f398aee1f91cdf2c44c7d8a0a8ca85d66c137360b68fe4f95ee9907a8 not found: ID does not exist"
Jul 29 14:45:16.235726 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:16.235702    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8bbd64c5-hpzxn"]
Jul 29 14:45:16.240056 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:16.240032    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-8bbd64c5-hpzxn"]
Jul 29 14:45:16.432236 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:16.432204    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b91a1c6-4e48-416b-8ccd-635f144f4168" path="/var/lib/kubelet/pods/5b91a1c6-4e48-416b-8ccd-635f144f4168/volumes"
Jul 29 14:45:36.392590 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:36.392505    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-55fcb658db-kl2k7"]
Jul 29 14:45:36.393172 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:36.392840    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b91a1c6-4e48-416b-8ccd-635f144f4168" containerName="manager"
Jul 29 14:45:36.393172 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:36.392851    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b91a1c6-4e48-416b-8ccd-635f144f4168" containerName="manager"
Jul 29 14:45:36.393172 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:36.392918    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b91a1c6-4e48-416b-8ccd-635f144f4168" containerName="manager"
Jul 29 14:45:36.400705 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:36.400682    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55fcb658db-kl2k7"
Jul 29 14:45:36.403101 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:36.403072    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 29 14:45:36.403243 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:36.403198    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hf9kj\""
Jul 29 14:45:36.405354 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:36.405329    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55fcb658db-kl2k7"]
Jul 29 14:45:36.522660 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:36.522629    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cc2924d-9b81-44dc-bf27-6bc6a7472b06-maas-api-tls\") pod \"maas-api-55fcb658db-kl2k7\" (UID: \"2cc2924d-9b81-44dc-bf27-6bc6a7472b06\") " pod="odh-ai-gateway-infra/maas-api-55fcb658db-kl2k7"
Jul 29 14:45:36.522858 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:36.522687    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cc2924d-9b81-44dc-bf27-6bc6a7472b06-openshift-service-ca\") pod \"maas-api-55fcb658db-kl2k7\" (UID: \"2cc2924d-9b81-44dc-bf27-6bc6a7472b06\") " pod="odh-ai-gateway-infra/maas-api-55fcb658db-kl2k7"
Jul 29 14:45:36.522858 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:36.522729    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59k9f\" (UniqueName: \"kubernetes.io/projected/2cc2924d-9b81-44dc-bf27-6bc6a7472b06-kube-api-access-59k9f\") pod \"maas-api-55fcb658db-kl2k7\" (UID: \"2cc2924d-9b81-44dc-bf27-6bc6a7472b06\") " pod="odh-ai-gateway-infra/maas-api-55fcb658db-kl2k7"
Jul 29 14:45:36.624099 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:36.624070    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cc2924d-9b81-44dc-bf27-6bc6a7472b06-maas-api-tls\") pod \"maas-api-55fcb658db-kl2k7\" (UID: \"2cc2924d-9b81-44dc-bf27-6bc6a7472b06\") " pod="odh-ai-gateway-infra/maas-api-55fcb658db-kl2k7"
Jul 29 14:45:36.624283 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:36.624112    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cc2924d-9b81-44dc-bf27-6bc6a7472b06-openshift-service-ca\") pod \"maas-api-55fcb658db-kl2k7\" (UID: \"2cc2924d-9b81-44dc-bf27-6bc6a7472b06\") " pod="odh-ai-gateway-infra/maas-api-55fcb658db-kl2k7"
Jul 29 14:45:36.624283 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:36.624213    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59k9f\" (UniqueName: \"kubernetes.io/projected/2cc2924d-9b81-44dc-bf27-6bc6a7472b06-kube-api-access-59k9f\") pod \"maas-api-55fcb658db-kl2k7\" (UID: \"2cc2924d-9b81-44dc-bf27-6bc6a7472b06\") " pod="odh-ai-gateway-infra/maas-api-55fcb658db-kl2k7"
Jul 29 14:45:36.624813 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:36.624792    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cc2924d-9b81-44dc-bf27-6bc6a7472b06-openshift-service-ca\") pod \"maas-api-55fcb658db-kl2k7\" (UID: \"2cc2924d-9b81-44dc-bf27-6bc6a7472b06\") " pod="odh-ai-gateway-infra/maas-api-55fcb658db-kl2k7"
Jul 29 14:45:36.626460 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:36.626439    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cc2924d-9b81-44dc-bf27-6bc6a7472b06-maas-api-tls\") pod \"maas-api-55fcb658db-kl2k7\" (UID: \"2cc2924d-9b81-44dc-bf27-6bc6a7472b06\") " pod="odh-ai-gateway-infra/maas-api-55fcb658db-kl2k7"
Jul 29 14:45:36.634975 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:36.634953    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59k9f\" (UniqueName: \"kubernetes.io/projected/2cc2924d-9b81-44dc-bf27-6bc6a7472b06-kube-api-access-59k9f\") pod \"maas-api-55fcb658db-kl2k7\" (UID: \"2cc2924d-9b81-44dc-bf27-6bc6a7472b06\") " pod="odh-ai-gateway-infra/maas-api-55fcb658db-kl2k7"
Jul 29 14:45:36.711762 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:36.711731    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55fcb658db-kl2k7"
Jul 29 14:45:36.844568 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:36.844425    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55fcb658db-kl2k7"]
Jul 29 14:45:36.847371 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:45:36.847331    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc2924d_9b81_44dc_bf27_6bc6a7472b06.slice/crio-f3ded65de05848ffdbf062e186a574a34079a696b6be87a9244ea610855f6f80 WatchSource:0}: Error finding container f3ded65de05848ffdbf062e186a574a34079a696b6be87a9244ea610855f6f80: Status 404 returned error can't find the container with id f3ded65de05848ffdbf062e186a574a34079a696b6be87a9244ea610855f6f80
Jul 29 14:45:37.302824 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:37.302788    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55fcb658db-kl2k7" event={"ID":"2cc2924d-9b81-44dc-bf27-6bc6a7472b06","Type":"ContainerStarted","Data":"f3ded65de05848ffdbf062e186a574a34079a696b6be87a9244ea610855f6f80"}
Jul 29 14:45:39.314504 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:39.314469    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55fcb658db-kl2k7" event={"ID":"2cc2924d-9b81-44dc-bf27-6bc6a7472b06","Type":"ContainerStarted","Data":"757315f205071fed3fc7d63358f7f678b6be43f3b1e16b928f4c70dce65e7edd"}
Jul 29 14:45:39.315158 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:39.314618    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-55fcb658db-kl2k7"
Jul 29 14:45:39.331547 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:39.331494    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-55fcb658db-kl2k7" podStartSLOduration=1.35458791 podStartE2EDuration="3.331476652s" totalImagesPullingTime="1.976888742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:45:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:45:39.329459203 +0000 UTC m=+641.377917155" watchObservedRunningTime="2026-07-29 14:45:39.331476652 +0000 UTC m=+641.379934646"
Jul 29 14:45:45.325991 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:45:45.325962    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-55fcb658db-kl2k7"
Jul 29 14:46:04.996496 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:04.996461    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2"]
Jul 29 14:46:05.001048 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:05.001029    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2"
Jul 29 14:46:05.003256 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:05.003232    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6c2gw\""
Jul 29 14:46:05.003365 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:05.003232    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 29 14:46:05.003824 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:05.003797    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 29 14:46:05.003980 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:05.003963    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 29 14:46:05.007902 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:05.007878    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2"]
Jul 29 14:46:05.067950 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:05.067913    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f545ede-4a2b-4e4d-996a-ec9628316097-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2\" (UID: \"1f545ede-4a2b-4e4d-996a-ec9628316097\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2"
Jul 29 14:46:05.068147 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:05.067955    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f545ede-4a2b-4e4d-996a-ec9628316097-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2\" (UID: \"1f545ede-4a2b-4e4d-996a-ec9628316097\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2"
Jul 29 14:46:05.068147 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:05.067992    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f545ede-4a2b-4e4d-996a-ec9628316097-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2\" (UID: \"1f545ede-4a2b-4e4d-996a-ec9628316097\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2"
Jul 29 14:46:05.068147 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:05.068044    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-562p6\" (UniqueName: \"kubernetes.io/projected/1f545ede-4a2b-4e4d-996a-ec9628316097-kube-api-access-562p6\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2\" (UID: \"1f545ede-4a2b-4e4d-996a-ec9628316097\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2"
Jul 29 14:46:05.068147 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:05.068099    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f545ede-4a2b-4e4d-996a-ec9628316097-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2\" (UID: \"1f545ede-4a2b-4e4d-996a-ec9628316097\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2"
Jul 29 14:46:05.068334 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:05.068195    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f545ede-4a2b-4e4d-996a-ec9628316097-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2\" (UID: \"1f545ede-4a2b-4e4d-996a-ec9628316097\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2"
Jul 29 14:46:05.168969 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:05.168928    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f545ede-4a2b-4e4d-996a-ec9628316097-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2\" (UID: \"1f545ede-4a2b-4e4d-996a-ec9628316097\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2"
Jul 29 14:46:05.169192 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:05.169021    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f545ede-4a2b-4e4d-996a-ec9628316097-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2\" (UID: \"1f545ede-4a2b-4e4d-996a-ec9628316097\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2"
Jul 29 14:46:05.169192 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:05.169049    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f545ede-4a2b-4e4d-996a-ec9628316097-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2\" (UID: \"1f545ede-4a2b-4e4d-996a-ec9628316097\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2"
Jul 29 14:46:05.169192 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:05.169073    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f545ede-4a2b-4e4d-996a-ec9628316097-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2\" (UID: \"1f545ede-4a2b-4e4d-996a-ec9628316097\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2"
Jul 29 14:46:05.169380 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:05.169286    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f545ede-4a2b-4e4d-996a-ec9628316097-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2\" (UID: \"1f545ede-4a2b-4e4d-996a-ec9628316097\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2"
Jul 29 14:46:05.169380 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:05.169354    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-562p6\" (UniqueName: \"kubernetes.io/projected/1f545ede-4a2b-4e4d-996a-ec9628316097-kube-api-access-562p6\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2\" (UID: \"1f545ede-4a2b-4e4d-996a-ec9628316097\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2"
Jul 29 14:46:05.169488 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:05.169399    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f545ede-4a2b-4e4d-996a-ec9628316097-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2\" (UID: \"1f545ede-4a2b-4e4d-996a-ec9628316097\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2"
Jul 29 14:46:05.169488 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:05.169410    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f545ede-4a2b-4e4d-996a-ec9628316097-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2\" (UID: \"1f545ede-4a2b-4e4d-996a-ec9628316097\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2"
Jul 29 14:46:05.169589 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:05.169561    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f545ede-4a2b-4e4d-996a-ec9628316097-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2\" (UID: \"1f545ede-4a2b-4e4d-996a-ec9628316097\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2"
Jul 29 14:46:05.171435 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:05.171408    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f545ede-4a2b-4e4d-996a-ec9628316097-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2\" (UID: \"1f545ede-4a2b-4e4d-996a-ec9628316097\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2"
Jul 29 14:46:05.171794 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:05.171776    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f545ede-4a2b-4e4d-996a-ec9628316097-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2\" (UID: \"1f545ede-4a2b-4e4d-996a-ec9628316097\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2"
Jul 29 14:46:05.176939 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:05.176919    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-562p6\" (UniqueName: \"kubernetes.io/projected/1f545ede-4a2b-4e4d-996a-ec9628316097-kube-api-access-562p6\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2\" (UID: \"1f545ede-4a2b-4e4d-996a-ec9628316097\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2"
Jul 29 14:46:05.312840 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:05.312756    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2"
Jul 29 14:46:05.443047 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:05.443022    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2"]
Jul 29 14:46:05.445732 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:46:05.445701    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f545ede_4a2b_4e4d_996a_ec9628316097.slice/crio-5b84dbfe61352a95446524acc5c2a3b68fc5bcc637e8f022684f5d7449653f7c WatchSource:0}: Error finding container 5b84dbfe61352a95446524acc5c2a3b68fc5bcc637e8f022684f5d7449653f7c: Status 404 returned error can't find the container with id 5b84dbfe61352a95446524acc5c2a3b68fc5bcc637e8f022684f5d7449653f7c
Jul 29 14:46:06.442803 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:06.442766    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2" event={"ID":"1f545ede-4a2b-4e4d-996a-ec9628316097","Type":"ContainerStarted","Data":"5b84dbfe61352a95446524acc5c2a3b68fc5bcc637e8f022684f5d7449653f7c"}
Jul 29 14:46:07.448856 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:07.448821    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2" event={"ID":"1f545ede-4a2b-4e4d-996a-ec9628316097","Type":"ContainerStarted","Data":"759e0a660bb7856733ec48718c0440bd983cfaecf06dd184540a72eff986cfa8"}
Jul 29 14:46:07.465774 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:07.465722    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2" podStartSLOduration=2.225619416 podStartE2EDuration="3.465706716s" totalImagesPullingTime="1.2400873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:46:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:46:07.463377334 +0000 UTC m=+669.511835275" watchObservedRunningTime="2026-07-29 14:46:07.465706716 +0000 UTC m=+669.514164655"
Jul 29 14:46:15.313335 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:15.313302    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2"
Jul 29 14:46:15.313978 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:15.313366    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2"
Jul 29 14:46:15.326244 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:15.326219    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2"
Jul 29 14:46:15.500636 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:15.500604    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2"
Jul 29 14:46:37.303690 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:37.303658    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms"]
Jul 29 14:46:37.309696 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:37.309677    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms"
Jul 29 14:46:37.311608 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:37.311586    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 29 14:46:37.314954 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:37.314932    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms"]
Jul 29 14:46:37.456183 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:37.456126    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b39f59c-2d08-479b-b89f-d9f47a342d5f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms\" (UID: \"5b39f59c-2d08-479b-b89f-d9f47a342d5f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms"
Jul 29 14:46:37.456341 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:37.456248    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnrdr\" (UniqueName: \"kubernetes.io/projected/5b39f59c-2d08-479b-b89f-d9f47a342d5f-kube-api-access-xnrdr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms\" (UID: \"5b39f59c-2d08-479b-b89f-d9f47a342d5f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms"
Jul 29 14:46:37.456389 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:37.456330    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b39f59c-2d08-479b-b89f-d9f47a342d5f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms\" (UID: \"5b39f59c-2d08-479b-b89f-d9f47a342d5f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms"
Jul 29 14:46:37.456389 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:37.456366    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b39f59c-2d08-479b-b89f-d9f47a342d5f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms\" (UID: \"5b39f59c-2d08-479b-b89f-d9f47a342d5f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms"
Jul 29 14:46:37.456458 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:37.456394    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b39f59c-2d08-479b-b89f-d9f47a342d5f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms\" (UID: \"5b39f59c-2d08-479b-b89f-d9f47a342d5f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms"
Jul 29 14:46:37.456458 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:37.456409    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b39f59c-2d08-479b-b89f-d9f47a342d5f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms\" (UID: \"5b39f59c-2d08-479b-b89f-d9f47a342d5f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms"
Jul 29 14:46:37.557777 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:37.557698    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b39f59c-2d08-479b-b89f-d9f47a342d5f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms\" (UID: \"5b39f59c-2d08-479b-b89f-d9f47a342d5f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms"
Jul 29 14:46:37.557777 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:37.557736    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b39f59c-2d08-479b-b89f-d9f47a342d5f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms\" (UID: \"5b39f59c-2d08-479b-b89f-d9f47a342d5f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms"
Jul 29 14:46:37.557777 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:37.557764    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b39f59c-2d08-479b-b89f-d9f47a342d5f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms\" (UID: \"5b39f59c-2d08-479b-b89f-d9f47a342d5f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms"
Jul 29 14:46:37.558013 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:37.557836    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnrdr\" (UniqueName: \"kubernetes.io/projected/5b39f59c-2d08-479b-b89f-d9f47a342d5f-kube-api-access-xnrdr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms\" (UID: \"5b39f59c-2d08-479b-b89f-d9f47a342d5f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms"
Jul 29 14:46:37.558013 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:37.557888    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b39f59c-2d08-479b-b89f-d9f47a342d5f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms\" (UID: \"5b39f59c-2d08-479b-b89f-d9f47a342d5f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms"
Jul 29 14:46:37.558013 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:37.557914    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b39f59c-2d08-479b-b89f-d9f47a342d5f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms\" (UID: \"5b39f59c-2d08-479b-b89f-d9f47a342d5f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms"
Jul 29 14:46:37.558203 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:37.558075    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b39f59c-2d08-479b-b89f-d9f47a342d5f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms\" (UID: \"5b39f59c-2d08-479b-b89f-d9f47a342d5f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms"
Jul 29 14:46:37.558203 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:37.558177    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b39f59c-2d08-479b-b89f-d9f47a342d5f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms\" (UID: \"5b39f59c-2d08-479b-b89f-d9f47a342d5f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms"
Jul 29 14:46:37.558319 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:37.558205    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b39f59c-2d08-479b-b89f-d9f47a342d5f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms\" (UID: \"5b39f59c-2d08-479b-b89f-d9f47a342d5f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms"
Jul 29 14:46:37.560121 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:37.560101    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b39f59c-2d08-479b-b89f-d9f47a342d5f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms\" (UID: \"5b39f59c-2d08-479b-b89f-d9f47a342d5f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms"
Jul 29 14:46:37.560311 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:37.560293    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b39f59c-2d08-479b-b89f-d9f47a342d5f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms\" (UID: \"5b39f59c-2d08-479b-b89f-d9f47a342d5f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms"
Jul 29 14:46:37.565371 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:37.565351    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnrdr\" (UniqueName: \"kubernetes.io/projected/5b39f59c-2d08-479b-b89f-d9f47a342d5f-kube-api-access-xnrdr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms\" (UID: \"5b39f59c-2d08-479b-b89f-d9f47a342d5f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms"
Jul 29 14:46:37.621341 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:37.621317    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms"
Jul 29 14:46:37.756691 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:37.756668    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms"]
Jul 29 14:46:37.759146 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:46:37.759095    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b39f59c_2d08_479b_b89f_d9f47a342d5f.slice/crio-84a384cc3f8c52fcab4d7de5d6c343328b8831fcb23afb437609fd7a67747a14 WatchSource:0}: Error finding container 84a384cc3f8c52fcab4d7de5d6c343328b8831fcb23afb437609fd7a67747a14: Status 404 returned error can't find the container with id 84a384cc3f8c52fcab4d7de5d6c343328b8831fcb23afb437609fd7a67747a14
Jul 29 14:46:37.760495 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:37.760476    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 14:46:38.597358 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:38.597319    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms" event={"ID":"5b39f59c-2d08-479b-b89f-d9f47a342d5f","Type":"ContainerStarted","Data":"e5a68608285e47a2a4282dbda57642860386ff2b8a0478dea81b8e3249c220ca"}
Jul 29 14:46:38.597749 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:38.597382    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms" event={"ID":"5b39f59c-2d08-479b-b89f-d9f47a342d5f","Type":"ContainerStarted","Data":"84a384cc3f8c52fcab4d7de5d6c343328b8831fcb23afb437609fd7a67747a14"}
Jul 29 14:46:38.615911 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:38.615866    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms" podStartSLOduration=1.395393723 podStartE2EDuration="1.615850015s" totalImagesPullingTime="220.456292ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:46:38.613657752 +0000 UTC m=+700.662115691" watchObservedRunningTime="2026-07-29 14:46:38.615850015 +0000 UTC m=+700.664307954"
Jul 29 14:46:47.622090 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:47.622048    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms"
Jul 29 14:46:47.622627 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:47.622124    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms"
Jul 29 14:46:47.634500 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:47.634475    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms"
Jul 29 14:46:47.649610 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:47.649587    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms"
Jul 29 14:46:51.007208 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:51.007116    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-47crv"]
Jul 29 14:46:51.011774 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:51.011741    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-47crv"
Jul 29 14:46:51.013655 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:51.013635    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 29 14:46:51.019870 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:51.019848    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-47crv"]
Jul 29 14:46:51.067508 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:51.067473    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93de9b1a-dbb7-4255-b4b3-8e96ca7e2b6c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-47crv\" (UID: \"93de9b1a-dbb7-4255-b4b3-8e96ca7e2b6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-47crv"
Jul 29 14:46:51.067671 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:51.067538    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93de9b1a-dbb7-4255-b4b3-8e96ca7e2b6c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-47crv\" (UID: \"93de9b1a-dbb7-4255-b4b3-8e96ca7e2b6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-47crv"
Jul 29 14:46:51.067671 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:51.067556    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6cf8\" (UniqueName: \"kubernetes.io/projected/93de9b1a-dbb7-4255-b4b3-8e96ca7e2b6c-kube-api-access-q6cf8\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-47crv\" (UID: \"93de9b1a-dbb7-4255-b4b3-8e96ca7e2b6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-47crv"
Jul 29 14:46:51.067671 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:51.067580    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93de9b1a-dbb7-4255-b4b3-8e96ca7e2b6c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-47crv\" (UID: \"93de9b1a-dbb7-4255-b4b3-8e96ca7e2b6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-47crv"
Jul 29 14:46:51.067671 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:51.067608    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93de9b1a-dbb7-4255-b4b3-8e96ca7e2b6c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-47crv\" (UID: \"93de9b1a-dbb7-4255-b4b3-8e96ca7e2b6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-47crv"
Jul 29 14:46:51.067826 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:51.067694    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93de9b1a-dbb7-4255-b4b3-8e96ca7e2b6c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-47crv\" (UID: \"93de9b1a-dbb7-4255-b4b3-8e96ca7e2b6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-47crv"
Jul 29 14:46:51.169081 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:51.169044    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93de9b1a-dbb7-4255-b4b3-8e96ca7e2b6c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-47crv\" (UID: \"93de9b1a-dbb7-4255-b4b3-8e96ca7e2b6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-47crv"
Jul 29 14:46:51.169273 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:51.169158    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93de9b1a-dbb7-4255-b4b3-8e96ca7e2b6c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-47crv\" (UID: \"93de9b1a-dbb7-4255-b4b3-8e96ca7e2b6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-47crv"
Jul 29 14:46:51.169273 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:51.169209    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93de9b1a-dbb7-4255-b4b3-8e96ca7e2b6c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-47crv\" (UID: \"93de9b1a-dbb7-4255-b4b3-8e96ca7e2b6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-47crv"
Jul 29 14:46:51.169379 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:51.169286    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93de9b1a-dbb7-4255-b4b3-8e96ca7e2b6c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-47crv\" (UID: \"93de9b1a-dbb7-4255-b4b3-8e96ca7e2b6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-47crv"
Jul 29 14:46:51.169379 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:51.169360    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93de9b1a-dbb7-4255-b4b3-8e96ca7e2b6c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-47crv\" (UID: \"93de9b1a-dbb7-4255-b4b3-8e96ca7e2b6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-47crv"
Jul 29 14:46:51.169480 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:51.169395    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6cf8\" (UniqueName: \"kubernetes.io/projected/93de9b1a-dbb7-4255-b4b3-8e96ca7e2b6c-kube-api-access-q6cf8\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-47crv\" (UID: \"93de9b1a-dbb7-4255-b4b3-8e96ca7e2b6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-47crv"
Jul 29 14:46:51.169480 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:51.169431    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93de9b1a-dbb7-4255-b4b3-8e96ca7e2b6c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-47crv\" (UID: \"93de9b1a-dbb7-4255-b4b3-8e96ca7e2b6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-47crv"
Jul 29 14:46:51.169630 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:51.169612    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93de9b1a-dbb7-4255-b4b3-8e96ca7e2b6c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-47crv\" (UID: \"93de9b1a-dbb7-4255-b4b3-8e96ca7e2b6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-47crv"
Jul 29 14:46:51.169691 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:51.169653    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93de9b1a-dbb7-4255-b4b3-8e96ca7e2b6c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-47crv\" (UID: \"93de9b1a-dbb7-4255-b4b3-8e96ca7e2b6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-47crv"
Jul 29 14:46:51.171405 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:51.171381    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93de9b1a-dbb7-4255-b4b3-8e96ca7e2b6c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-47crv\" (UID: \"93de9b1a-dbb7-4255-b4b3-8e96ca7e2b6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-47crv"
Jul 29 14:46:51.171952 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:51.171932    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93de9b1a-dbb7-4255-b4b3-8e96ca7e2b6c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-47crv\" (UID: \"93de9b1a-dbb7-4255-b4b3-8e96ca7e2b6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-47crv"
Jul 29 14:46:51.176767 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:51.176745    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6cf8\" (UniqueName: \"kubernetes.io/projected/93de9b1a-dbb7-4255-b4b3-8e96ca7e2b6c-kube-api-access-q6cf8\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-47crv\" (UID: \"93de9b1a-dbb7-4255-b4b3-8e96ca7e2b6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-47crv"
Jul 29 14:46:51.323387 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:51.323300    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-47crv"
Jul 29 14:46:51.456144 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:51.456102    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-47crv"]
Jul 29 14:46:51.459798 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:46:51.459773    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93de9b1a_dbb7_4255_b4b3_8e96ca7e2b6c.slice/crio-b96a8a0d33c676e6648937cf8b0b44aa3a9366f7602dadec51360c6c39ee1ac2 WatchSource:0}: Error finding container b96a8a0d33c676e6648937cf8b0b44aa3a9366f7602dadec51360c6c39ee1ac2: Status 404 returned error can't find the container with id b96a8a0d33c676e6648937cf8b0b44aa3a9366f7602dadec51360c6c39ee1ac2
Jul 29 14:46:51.660227 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:51.660148    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-47crv" event={"ID":"93de9b1a-dbb7-4255-b4b3-8e96ca7e2b6c","Type":"ContainerStarted","Data":"b96a8a0d33c676e6648937cf8b0b44aa3a9366f7602dadec51360c6c39ee1ac2"}
Jul 29 14:46:52.666548 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:52.666510    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-47crv" event={"ID":"93de9b1a-dbb7-4255-b4b3-8e96ca7e2b6c","Type":"ContainerStarted","Data":"125d035c799b1fe7109001927b3ce442cf5763ac53fb92a8cadc25152a15ea8d"}
Jul 29 14:46:52.683211 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:46:52.683166    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-47crv" podStartSLOduration=2.456298608 podStartE2EDuration="2.68315262s" totalImagesPullingTime="226.854012ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:46:52.682205124 +0000 UTC m=+714.730663067" watchObservedRunningTime="2026-07-29 14:46:52.68315262 +0000 UTC m=+714.731610550"
Jul 29 14:47:01.323613 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:47:01.323569    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-47crv"
Jul 29 14:47:01.324100 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:47:01.323645    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-47crv"
Jul 29 14:47:01.336588 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:47:01.336561    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-47crv"
Jul 29 14:47:01.721836 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:47:01.721807    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-47crv"
Jul 29 14:47:40.402973 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:47:40.402925    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-66b68bbbfd-vlhgs"]
Jul 29 14:47:40.403565 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:47:40.403113    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-66b68bbbfd-vlhgs" podUID="9e97e830-b9a1-467a-95b4-08ddd21c28c7" containerName="authorino" containerID="cri-o://4c9873a203662df73c80c9ea444dbc7c9ebfa4993b4fc5703f71224235ab1f8c" gracePeriod=30
Jul 29 14:47:40.650278 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:47:40.650256    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66b68bbbfd-vlhgs"
Jul 29 14:47:40.711293 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:47:40.711264    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82d6v\" (UniqueName: \"kubernetes.io/projected/9e97e830-b9a1-467a-95b4-08ddd21c28c7-kube-api-access-82d6v\") pod \"9e97e830-b9a1-467a-95b4-08ddd21c28c7\" (UID: \"9e97e830-b9a1-467a-95b4-08ddd21c28c7\") "
Jul 29 14:47:40.711433 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:47:40.711308    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9e97e830-b9a1-467a-95b4-08ddd21c28c7-tls-cert\") pod \"9e97e830-b9a1-467a-95b4-08ddd21c28c7\" (UID: \"9e97e830-b9a1-467a-95b4-08ddd21c28c7\") "
Jul 29 14:47:40.713385 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:47:40.713359    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e97e830-b9a1-467a-95b4-08ddd21c28c7-kube-api-access-82d6v" (OuterVolumeSpecName: "kube-api-access-82d6v") pod "9e97e830-b9a1-467a-95b4-08ddd21c28c7" (UID: "9e97e830-b9a1-467a-95b4-08ddd21c28c7"). InnerVolumeSpecName "kube-api-access-82d6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 14:47:40.723112 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:47:40.723090    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e97e830-b9a1-467a-95b4-08ddd21c28c7-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "9e97e830-b9a1-467a-95b4-08ddd21c28c7" (UID: "9e97e830-b9a1-467a-95b4-08ddd21c28c7"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 14:47:40.812740 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:47:40.812699    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82d6v\" (UniqueName: \"kubernetes.io/projected/9e97e830-b9a1-467a-95b4-08ddd21c28c7-kube-api-access-82d6v\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 14:47:40.812740 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:47:40.812732    2563 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9e97e830-b9a1-467a-95b4-08ddd21c28c7-tls-cert\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 14:47:40.894516 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:47:40.894476    2563 generic.go:356] "Generic (PLEG): container finished" podID="9e97e830-b9a1-467a-95b4-08ddd21c28c7" containerID="4c9873a203662df73c80c9ea444dbc7c9ebfa4993b4fc5703f71224235ab1f8c" exitCode=0
Jul 29 14:47:40.894516 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:47:40.894520    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66b68bbbfd-vlhgs"
Jul 29 14:47:40.894715 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:47:40.894525    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66b68bbbfd-vlhgs" event={"ID":"9e97e830-b9a1-467a-95b4-08ddd21c28c7","Type":"ContainerDied","Data":"4c9873a203662df73c80c9ea444dbc7c9ebfa4993b4fc5703f71224235ab1f8c"}
Jul 29 14:47:40.894715 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:47:40.894574    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66b68bbbfd-vlhgs" event={"ID":"9e97e830-b9a1-467a-95b4-08ddd21c28c7","Type":"ContainerDied","Data":"ff618b97e1649ee7f64e422f25630b740b9bc2d55b831d707728fb6235ddde29"}
Jul 29 14:47:40.894715 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:47:40.894595    2563 scope.go:117] "RemoveContainer" containerID="4c9873a203662df73c80c9ea444dbc7c9ebfa4993b4fc5703f71224235ab1f8c"
Jul 29 14:47:40.903543 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:47:40.903524    2563 scope.go:117] "RemoveContainer" containerID="4c9873a203662df73c80c9ea444dbc7c9ebfa4993b4fc5703f71224235ab1f8c"
Jul 29 14:47:40.903776 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:47:40.903757    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c9873a203662df73c80c9ea444dbc7c9ebfa4993b4fc5703f71224235ab1f8c\": container with ID starting with 4c9873a203662df73c80c9ea444dbc7c9ebfa4993b4fc5703f71224235ab1f8c not found: ID does not exist" containerID="4c9873a203662df73c80c9ea444dbc7c9ebfa4993b4fc5703f71224235ab1f8c"
Jul 29 14:47:40.903820 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:47:40.903784    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c9873a203662df73c80c9ea444dbc7c9ebfa4993b4fc5703f71224235ab1f8c"} err="failed to get container status \"4c9873a203662df73c80c9ea444dbc7c9ebfa4993b4fc5703f71224235ab1f8c\": rpc error: code = NotFound desc = could not find container \"4c9873a203662df73c80c9ea444dbc7c9ebfa4993b4fc5703f71224235ab1f8c\": container with ID starting with 4c9873a203662df73c80c9ea444dbc7c9ebfa4993b4fc5703f71224235ab1f8c not found: ID does not exist"
Jul 29 14:47:40.914525 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:47:40.914502    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-66b68bbbfd-vlhgs"]
Jul 29 14:47:40.916511 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:47:40.916491    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-66b68bbbfd-vlhgs"]
Jul 29 14:47:42.432831 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:47:42.432795    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e97e830-b9a1-467a-95b4-08ddd21c28c7" path="/var/lib/kubelet/pods/9e97e830-b9a1-467a-95b4-08ddd21c28c7/volumes"
Jul 29 14:48:26.575052 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:48:26.574968    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-794c9d7b98-g46sn"]
Jul 29 14:48:26.575611 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:48:26.575214    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-794c9d7b98-g46sn" podUID="4cc52c77-cd0d-415c-8bce-c4132bfe66f2" containerName="manager" containerID="cri-o://f1e6bc45a36b432660c84e15ecd314f97c6f067a47aa2e4dbb9f1884a6a50479" gracePeriod=30
Jul 29 14:48:26.829453 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:48:26.829395    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-794c9d7b98-g46sn"
Jul 29 14:48:26.894391 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:48:26.894354    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzh48\" (UniqueName: \"kubernetes.io/projected/4cc52c77-cd0d-415c-8bce-c4132bfe66f2-kube-api-access-fzh48\") pod \"4cc52c77-cd0d-415c-8bce-c4132bfe66f2\" (UID: \"4cc52c77-cd0d-415c-8bce-c4132bfe66f2\") "
Jul 29 14:48:26.894554 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:48:26.894439    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4cc52c77-cd0d-415c-8bce-c4132bfe66f2-webhook-cert\") pod \"4cc52c77-cd0d-415c-8bce-c4132bfe66f2\" (UID: \"4cc52c77-cd0d-415c-8bce-c4132bfe66f2\") "
Jul 29 14:48:26.896382 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:48:26.896349    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cc52c77-cd0d-415c-8bce-c4132bfe66f2-kube-api-access-fzh48" (OuterVolumeSpecName: "kube-api-access-fzh48") pod "4cc52c77-cd0d-415c-8bce-c4132bfe66f2" (UID: "4cc52c77-cd0d-415c-8bce-c4132bfe66f2"). InnerVolumeSpecName "kube-api-access-fzh48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 14:48:26.896382 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:48:26.896366    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cc52c77-cd0d-415c-8bce-c4132bfe66f2-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "4cc52c77-cd0d-415c-8bce-c4132bfe66f2" (UID: "4cc52c77-cd0d-415c-8bce-c4132bfe66f2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 14:48:26.995277 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:48:26.995251    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzh48\" (UniqueName: \"kubernetes.io/projected/4cc52c77-cd0d-415c-8bce-c4132bfe66f2-kube-api-access-fzh48\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 14:48:26.995277 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:48:26.995275    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4cc52c77-cd0d-415c-8bce-c4132bfe66f2-webhook-cert\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 14:48:27.115027 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:48:27.114950    2563 generic.go:356] "Generic (PLEG): container finished" podID="4cc52c77-cd0d-415c-8bce-c4132bfe66f2" containerID="f1e6bc45a36b432660c84e15ecd314f97c6f067a47aa2e4dbb9f1884a6a50479" exitCode=0
Jul 29 14:48:27.115027 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:48:27.114993    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-794c9d7b98-g46sn"
Jul 29 14:48:27.115197 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:48:27.115037    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-794c9d7b98-g46sn" event={"ID":"4cc52c77-cd0d-415c-8bce-c4132bfe66f2","Type":"ContainerDied","Data":"f1e6bc45a36b432660c84e15ecd314f97c6f067a47aa2e4dbb9f1884a6a50479"}
Jul 29 14:48:27.115197 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:48:27.115090    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-794c9d7b98-g46sn" event={"ID":"4cc52c77-cd0d-415c-8bce-c4132bfe66f2","Type":"ContainerDied","Data":"ed404e68fb2c1bc81306eafae867debbfdc5e50e2d8def175a61a77ba81f5ba7"}
Jul 29 14:48:27.115197 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:48:27.115112    2563 scope.go:117] "RemoveContainer" containerID="f1e6bc45a36b432660c84e15ecd314f97c6f067a47aa2e4dbb9f1884a6a50479"
Jul 29 14:48:27.123831 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:48:27.123814    2563 scope.go:117] "RemoveContainer" containerID="f1e6bc45a36b432660c84e15ecd314f97c6f067a47aa2e4dbb9f1884a6a50479"
Jul 29 14:48:27.124071 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:48:27.124053    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1e6bc45a36b432660c84e15ecd314f97c6f067a47aa2e4dbb9f1884a6a50479\": container with ID starting with f1e6bc45a36b432660c84e15ecd314f97c6f067a47aa2e4dbb9f1884a6a50479 not found: ID does not exist" containerID="f1e6bc45a36b432660c84e15ecd314f97c6f067a47aa2e4dbb9f1884a6a50479"
Jul 29 14:48:27.124118 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:48:27.124078    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1e6bc45a36b432660c84e15ecd314f97c6f067a47aa2e4dbb9f1884a6a50479"} err="failed to get container status \"f1e6bc45a36b432660c84e15ecd314f97c6f067a47aa2e4dbb9f1884a6a50479\": rpc error: code = NotFound desc = could not find container \"f1e6bc45a36b432660c84e15ecd314f97c6f067a47aa2e4dbb9f1884a6a50479\": container with ID starting with f1e6bc45a36b432660c84e15ecd314f97c6f067a47aa2e4dbb9f1884a6a50479 not found: ID does not exist"
Jul 29 14:48:27.138260 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:48:27.138237    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-794c9d7b98-g46sn"]
Jul 29 14:48:27.146770 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:48:27.146748    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-794c9d7b98-g46sn"]
Jul 29 14:48:28.432387 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:48:28.432351    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cc52c77-cd0d-415c-8bce-c4132bfe66f2" path="/var/lib/kubelet/pods/4cc52c77-cd0d-415c-8bce-c4132bfe66f2/volumes"
Jul 29 14:49:27.947804 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:27.947769    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-794c9d7b98-s92db"]
Jul 29 14:49:27.948290 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:27.948103    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e97e830-b9a1-467a-95b4-08ddd21c28c7" containerName="authorino"
Jul 29 14:49:27.948290 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:27.948114    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e97e830-b9a1-467a-95b4-08ddd21c28c7" containerName="authorino"
Jul 29 14:49:27.948290 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:27.948150    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cc52c77-cd0d-415c-8bce-c4132bfe66f2" containerName="manager"
Jul 29 14:49:27.948290 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:27.948155    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cc52c77-cd0d-415c-8bce-c4132bfe66f2" containerName="manager"
Jul 29 14:49:27.948290 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:27.948210    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e97e830-b9a1-467a-95b4-08ddd21c28c7" containerName="authorino"
Jul 29 14:49:27.948290 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:27.948221    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cc52c77-cd0d-415c-8bce-c4132bfe66f2" containerName="manager"
Jul 29 14:49:27.951217 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:27.951201    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-794c9d7b98-s92db"
Jul 29 14:49:27.953676 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:27.953653    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-57frt\""
Jul 29 14:49:27.954115 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:27.954097    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 29 14:49:27.954174 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:27.954126    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 29 14:49:27.959774 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:27.959751    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-794c9d7b98-s92db"]
Jul 29 14:49:28.090611 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:28.090579    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddhft\" (UniqueName: \"kubernetes.io/projected/6f1afb74-d40c-4c1e-9138-848171c3bea2-kube-api-access-ddhft\") pod \"maas-controller-794c9d7b98-s92db\" (UID: \"6f1afb74-d40c-4c1e-9138-848171c3bea2\") " pod="opendatahub/maas-controller-794c9d7b98-s92db"
Jul 29 14:49:28.090611 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:28.090618    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6f1afb74-d40c-4c1e-9138-848171c3bea2-webhook-cert\") pod \"maas-controller-794c9d7b98-s92db\" (UID: \"6f1afb74-d40c-4c1e-9138-848171c3bea2\") " pod="opendatahub/maas-controller-794c9d7b98-s92db"
Jul 29 14:49:28.191489 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:28.191455    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddhft\" (UniqueName: \"kubernetes.io/projected/6f1afb74-d40c-4c1e-9138-848171c3bea2-kube-api-access-ddhft\") pod \"maas-controller-794c9d7b98-s92db\" (UID: \"6f1afb74-d40c-4c1e-9138-848171c3bea2\") " pod="opendatahub/maas-controller-794c9d7b98-s92db"
Jul 29 14:49:28.191643 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:28.191497    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6f1afb74-d40c-4c1e-9138-848171c3bea2-webhook-cert\") pod \"maas-controller-794c9d7b98-s92db\" (UID: \"6f1afb74-d40c-4c1e-9138-848171c3bea2\") " pod="opendatahub/maas-controller-794c9d7b98-s92db"
Jul 29 14:49:28.193886 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:28.193859    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6f1afb74-d40c-4c1e-9138-848171c3bea2-webhook-cert\") pod \"maas-controller-794c9d7b98-s92db\" (UID: \"6f1afb74-d40c-4c1e-9138-848171c3bea2\") " pod="opendatahub/maas-controller-794c9d7b98-s92db"
Jul 29 14:49:28.199602 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:28.199545    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddhft\" (UniqueName: \"kubernetes.io/projected/6f1afb74-d40c-4c1e-9138-848171c3bea2-kube-api-access-ddhft\") pod \"maas-controller-794c9d7b98-s92db\" (UID: \"6f1afb74-d40c-4c1e-9138-848171c3bea2\") " pod="opendatahub/maas-controller-794c9d7b98-s92db"
Jul 29 14:49:28.263697 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:28.263664    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-794c9d7b98-s92db"
Jul 29 14:49:28.393161 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:28.393103    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-794c9d7b98-s92db"]
Jul 29 14:49:28.396233 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:49:28.396205    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f1afb74_d40c_4c1e_9138_848171c3bea2.slice/crio-68c2dee9e669b6f1e9a198cdd2adabf8c3121c90da8a3b60eb4c2c70540f36a6 WatchSource:0}: Error finding container 68c2dee9e669b6f1e9a198cdd2adabf8c3121c90da8a3b60eb4c2c70540f36a6: Status 404 returned error can't find the container with id 68c2dee9e669b6f1e9a198cdd2adabf8c3121c90da8a3b60eb4c2c70540f36a6
Jul 29 14:49:29.402958 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:29.402921    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-794c9d7b98-s92db" event={"ID":"6f1afb74-d40c-4c1e-9138-848171c3bea2","Type":"ContainerStarted","Data":"3516a135ea63cd2eb503b98bb092fd7197844fcd99d1bafd2f5bf9e35207166d"}
Jul 29 14:49:29.403419 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:29.402972    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-794c9d7b98-s92db" event={"ID":"6f1afb74-d40c-4c1e-9138-848171c3bea2","Type":"ContainerStarted","Data":"68c2dee9e669b6f1e9a198cdd2adabf8c3121c90da8a3b60eb4c2c70540f36a6"}
Jul 29 14:49:29.403419 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:29.403020    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-794c9d7b98-s92db"
Jul 29 14:49:29.420864 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:29.420816    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-794c9d7b98-s92db" podStartSLOduration=1.953798107 podStartE2EDuration="2.420799608s" totalImagesPullingTime="467.001501ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:49:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:49:29.418925992 +0000 UTC m=+871.467383931" watchObservedRunningTime="2026-07-29 14:49:29.420799608 +0000 UTC m=+871.469257564"
Jul 29 14:49:40.409176 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:40.409123    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-794c9d7b98-s92db"
Jul 29 14:49:45.057780 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:45.057744    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-794c9d7b98-s92db"]
Jul 29 14:49:45.058191 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:45.057946    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-794c9d7b98-s92db" podUID="6f1afb74-d40c-4c1e-9138-848171c3bea2" containerName="manager" containerID="cri-o://3516a135ea63cd2eb503b98bb092fd7197844fcd99d1bafd2f5bf9e35207166d" gracePeriod=30
Jul 29 14:49:45.300481 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:45.300458    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-794c9d7b98-s92db"
Jul 29 14:49:45.336713 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:45.336652    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddhft\" (UniqueName: \"kubernetes.io/projected/6f1afb74-d40c-4c1e-9138-848171c3bea2-kube-api-access-ddhft\") pod \"6f1afb74-d40c-4c1e-9138-848171c3bea2\" (UID: \"6f1afb74-d40c-4c1e-9138-848171c3bea2\") "
Jul 29 14:49:45.336834 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:45.336751    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6f1afb74-d40c-4c1e-9138-848171c3bea2-webhook-cert\") pod \"6f1afb74-d40c-4c1e-9138-848171c3bea2\" (UID: \"6f1afb74-d40c-4c1e-9138-848171c3bea2\") "
Jul 29 14:49:45.338670 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:45.338643    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f1afb74-d40c-4c1e-9138-848171c3bea2-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "6f1afb74-d40c-4c1e-9138-848171c3bea2" (UID: "6f1afb74-d40c-4c1e-9138-848171c3bea2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 14:49:45.339050 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:45.339017    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f1afb74-d40c-4c1e-9138-848171c3bea2-kube-api-access-ddhft" (OuterVolumeSpecName: "kube-api-access-ddhft") pod "6f1afb74-d40c-4c1e-9138-848171c3bea2" (UID: "6f1afb74-d40c-4c1e-9138-848171c3bea2"). InnerVolumeSpecName "kube-api-access-ddhft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 14:49:45.438011 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:45.437966    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6f1afb74-d40c-4c1e-9138-848171c3bea2-webhook-cert\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 14:49:45.438011 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:45.438003    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddhft\" (UniqueName: \"kubernetes.io/projected/6f1afb74-d40c-4c1e-9138-848171c3bea2-kube-api-access-ddhft\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 14:49:45.473644 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:45.473613    2563 generic.go:356] "Generic (PLEG): container finished" podID="6f1afb74-d40c-4c1e-9138-848171c3bea2" containerID="3516a135ea63cd2eb503b98bb092fd7197844fcd99d1bafd2f5bf9e35207166d" exitCode=0
Jul 29 14:49:45.473786 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:45.473651    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-794c9d7b98-s92db"
Jul 29 14:49:45.473786 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:45.473680    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-794c9d7b98-s92db" event={"ID":"6f1afb74-d40c-4c1e-9138-848171c3bea2","Type":"ContainerDied","Data":"3516a135ea63cd2eb503b98bb092fd7197844fcd99d1bafd2f5bf9e35207166d"}
Jul 29 14:49:45.473786 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:45.473732    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-794c9d7b98-s92db" event={"ID":"6f1afb74-d40c-4c1e-9138-848171c3bea2","Type":"ContainerDied","Data":"68c2dee9e669b6f1e9a198cdd2adabf8c3121c90da8a3b60eb4c2c70540f36a6"}
Jul 29 14:49:45.473786 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:45.473764    2563 scope.go:117] "RemoveContainer" containerID="3516a135ea63cd2eb503b98bb092fd7197844fcd99d1bafd2f5bf9e35207166d"
Jul 29 14:49:45.482862 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:45.482846    2563 scope.go:117] "RemoveContainer" containerID="3516a135ea63cd2eb503b98bb092fd7197844fcd99d1bafd2f5bf9e35207166d"
Jul 29 14:49:45.483145 ip-10-0-136-196 kubenswrapper[2563]: E0729 14:49:45.483106    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3516a135ea63cd2eb503b98bb092fd7197844fcd99d1bafd2f5bf9e35207166d\": container with ID starting with 3516a135ea63cd2eb503b98bb092fd7197844fcd99d1bafd2f5bf9e35207166d not found: ID does not exist" containerID="3516a135ea63cd2eb503b98bb092fd7197844fcd99d1bafd2f5bf9e35207166d"
Jul 29 14:49:45.483237 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:45.483151    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3516a135ea63cd2eb503b98bb092fd7197844fcd99d1bafd2f5bf9e35207166d"} err="failed to get container status \"3516a135ea63cd2eb503b98bb092fd7197844fcd99d1bafd2f5bf9e35207166d\": rpc error: code = NotFound desc = could not find container \"3516a135ea63cd2eb503b98bb092fd7197844fcd99d1bafd2f5bf9e35207166d\": container with ID starting with 3516a135ea63cd2eb503b98bb092fd7197844fcd99d1bafd2f5bf9e35207166d not found: ID does not exist"
Jul 29 14:49:45.494337 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:45.494317    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-794c9d7b98-s92db"]
Jul 29 14:49:45.498428 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:45.498407    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-794c9d7b98-s92db"]
Jul 29 14:49:46.286243 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:46.286205    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-794c9d7b98-w5xq8"]
Jul 29 14:49:46.286611 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:46.286595    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f1afb74-d40c-4c1e-9138-848171c3bea2" containerName="manager"
Jul 29 14:49:46.286611 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:46.286608    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f1afb74-d40c-4c1e-9138-848171c3bea2" containerName="manager"
Jul 29 14:49:46.286684 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:46.286665    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f1afb74-d40c-4c1e-9138-848171c3bea2" containerName="manager"
Jul 29 14:49:46.290771 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:46.290754    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-794c9d7b98-w5xq8"
Jul 29 14:49:46.293316 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:46.293293    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 29 14:49:46.293316 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:46.293312    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-57frt\""
Jul 29 14:49:46.293441 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:46.293298    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 29 14:49:46.296948 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:46.296925    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-794c9d7b98-w5xq8"]
Jul 29 14:49:46.346347 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:46.346281    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d0858163-11e4-4dec-bd96-ca7ceb51e90e-webhook-cert\") pod \"maas-controller-794c9d7b98-w5xq8\" (UID: \"d0858163-11e4-4dec-bd96-ca7ceb51e90e\") " pod="opendatahub/maas-controller-794c9d7b98-w5xq8"
Jul 29 14:49:46.346347 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:46.346311    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bntb\" (UniqueName: \"kubernetes.io/projected/d0858163-11e4-4dec-bd96-ca7ceb51e90e-kube-api-access-9bntb\") pod \"maas-controller-794c9d7b98-w5xq8\" (UID: \"d0858163-11e4-4dec-bd96-ca7ceb51e90e\") " pod="opendatahub/maas-controller-794c9d7b98-w5xq8"
Jul 29 14:49:46.432921 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:46.432891    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f1afb74-d40c-4c1e-9138-848171c3bea2" path="/var/lib/kubelet/pods/6f1afb74-d40c-4c1e-9138-848171c3bea2/volumes"
Jul 29 14:49:46.447739 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:46.447715    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d0858163-11e4-4dec-bd96-ca7ceb51e90e-webhook-cert\") pod \"maas-controller-794c9d7b98-w5xq8\" (UID: \"d0858163-11e4-4dec-bd96-ca7ceb51e90e\") " pod="opendatahub/maas-controller-794c9d7b98-w5xq8"
Jul 29 14:49:46.447806 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:46.447747    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bntb\" (UniqueName: \"kubernetes.io/projected/d0858163-11e4-4dec-bd96-ca7ceb51e90e-kube-api-access-9bntb\") pod \"maas-controller-794c9d7b98-w5xq8\" (UID: \"d0858163-11e4-4dec-bd96-ca7ceb51e90e\") " pod="opendatahub/maas-controller-794c9d7b98-w5xq8"
Jul 29 14:49:46.450117 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:46.450084    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d0858163-11e4-4dec-bd96-ca7ceb51e90e-webhook-cert\") pod \"maas-controller-794c9d7b98-w5xq8\" (UID: \"d0858163-11e4-4dec-bd96-ca7ceb51e90e\") " pod="opendatahub/maas-controller-794c9d7b98-w5xq8"
Jul 29 14:49:46.455448 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:46.455428    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bntb\" (UniqueName: \"kubernetes.io/projected/d0858163-11e4-4dec-bd96-ca7ceb51e90e-kube-api-access-9bntb\") pod \"maas-controller-794c9d7b98-w5xq8\" (UID: \"d0858163-11e4-4dec-bd96-ca7ceb51e90e\") " pod="opendatahub/maas-controller-794c9d7b98-w5xq8"
Jul 29 14:49:46.602840 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:46.602780    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-794c9d7b98-w5xq8"
Jul 29 14:49:46.938215 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:46.937986    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-794c9d7b98-w5xq8"]
Jul 29 14:49:46.940741 ip-10-0-136-196 kubenswrapper[2563]: W0729 14:49:46.940718    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0858163_11e4_4dec_bd96_ca7ceb51e90e.slice/crio-67bcf6fa4102a1f19655297a855db29b00660c4c449226f86bac38f5c6477ae9 WatchSource:0}: Error finding container 67bcf6fa4102a1f19655297a855db29b00660c4c449226f86bac38f5c6477ae9: Status 404 returned error can't find the container with id 67bcf6fa4102a1f19655297a855db29b00660c4c449226f86bac38f5c6477ae9
Jul 29 14:49:47.484841 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:47.484810    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-794c9d7b98-w5xq8" event={"ID":"d0858163-11e4-4dec-bd96-ca7ceb51e90e","Type":"ContainerStarted","Data":"67bcf6fa4102a1f19655297a855db29b00660c4c449226f86bac38f5c6477ae9"}
Jul 29 14:49:48.490550 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:48.490514    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-794c9d7b98-w5xq8" event={"ID":"d0858163-11e4-4dec-bd96-ca7ceb51e90e","Type":"ContainerStarted","Data":"19d2198fb251d3bca948ec592cd147d935a42696a61fa326cfe6f6842fd56674"}
Jul 29 14:49:48.490909 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:48.490590    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-794c9d7b98-w5xq8"
Jul 29 14:49:48.509283 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:48.509232    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-794c9d7b98-w5xq8" podStartSLOduration=1.810325428 podStartE2EDuration="2.509217543s" totalImagesPullingTime="698.892115ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 14:49:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 14:49:48.505974 +0000 UTC m=+890.554431938" watchObservedRunningTime="2026-07-29 14:49:48.509217543 +0000 UTC m=+890.557675481"
Jul 29 14:49:58.387577 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:58.387508    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xnm95_23d79588-937d-4c6d-8760-a081024f608e/console-operator/1.log"
Jul 29 14:49:58.394859 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:58.394837    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xnm95_23d79588-937d-4c6d-8760-a081024f608e/console-operator/1.log"
Jul 29 14:49:59.500013 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:49:59.499981    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-794c9d7b98-w5xq8"
Jul 29 14:54:58.416862 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:54:58.416836    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xnm95_23d79588-937d-4c6d-8760-a081024f608e/console-operator/1.log"
Jul 29 14:54:58.421636 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:54:58.421614    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xnm95_23d79588-937d-4c6d-8760-a081024f608e/console-operator/1.log"
Jul 29 14:59:58.448022 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:59:58.447996    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xnm95_23d79588-937d-4c6d-8760-a081024f608e/console-operator/1.log"
Jul 29 14:59:58.448725 ip-10-0-136-196 kubenswrapper[2563]: I0729 14:59:58.448707    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xnm95_23d79588-937d-4c6d-8760-a081024f608e/console-operator/1.log"
Jul 29 15:03:01.955904 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:01.955867    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-59092fc4-79f7dcddff-l75hd"]
Jul 29 15:03:01.959883 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:01.959857    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59092fc4-79f7dcddff-l75hd"
Jul 29 15:03:01.962201 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:01.962182    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-59092fc4-serving-cert\""
Jul 29 15:03:01.973730 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:01.973702    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-59092fc4-79f7dcddff-l75hd"]
Jul 29 15:03:02.059866 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:02.059829    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-splws\" (UniqueName: \"kubernetes.io/projected/21f8e6a1-cf31-4fbf-985f-cdda8c238bbb-kube-api-access-splws\") pod \"maas-api-e2e-ait-59092fc4-79f7dcddff-l75hd\" (UID: \"21f8e6a1-cf31-4fbf-985f-cdda8c238bbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59092fc4-79f7dcddff-l75hd"
Jul 29 15:03:02.060057 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:02.059886    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21f8e6a1-cf31-4fbf-985f-cdda8c238bbb-maas-api-tls\") pod \"maas-api-e2e-ait-59092fc4-79f7dcddff-l75hd\" (UID: \"21f8e6a1-cf31-4fbf-985f-cdda8c238bbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59092fc4-79f7dcddff-l75hd"
Jul 29 15:03:02.060057 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:02.059953    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21f8e6a1-cf31-4fbf-985f-cdda8c238bbb-openshift-service-ca\") pod \"maas-api-e2e-ait-59092fc4-79f7dcddff-l75hd\" (UID: \"21f8e6a1-cf31-4fbf-985f-cdda8c238bbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59092fc4-79f7dcddff-l75hd"
Jul 29 15:03:02.161237 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:02.161207    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-splws\" (UniqueName: \"kubernetes.io/projected/21f8e6a1-cf31-4fbf-985f-cdda8c238bbb-kube-api-access-splws\") pod \"maas-api-e2e-ait-59092fc4-79f7dcddff-l75hd\" (UID: \"21f8e6a1-cf31-4fbf-985f-cdda8c238bbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59092fc4-79f7dcddff-l75hd"
Jul 29 15:03:02.161410 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:02.161252    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21f8e6a1-cf31-4fbf-985f-cdda8c238bbb-maas-api-tls\") pod \"maas-api-e2e-ait-59092fc4-79f7dcddff-l75hd\" (UID: \"21f8e6a1-cf31-4fbf-985f-cdda8c238bbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59092fc4-79f7dcddff-l75hd"
Jul 29 15:03:02.161410 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:02.161304    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21f8e6a1-cf31-4fbf-985f-cdda8c238bbb-openshift-service-ca\") pod \"maas-api-e2e-ait-59092fc4-79f7dcddff-l75hd\" (UID: \"21f8e6a1-cf31-4fbf-985f-cdda8c238bbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59092fc4-79f7dcddff-l75hd"
Jul 29 15:03:02.161482 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:03:02.161430    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-59092fc4-serving-cert: secret "maas-api-e2e-ait-59092fc4-serving-cert" not found
Jul 29 15:03:02.161521 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:03:02.161500    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21f8e6a1-cf31-4fbf-985f-cdda8c238bbb-maas-api-tls podName:21f8e6a1-cf31-4fbf-985f-cdda8c238bbb nodeName:}" failed. No retries permitted until 2026-07-29 15:03:02.661478828 +0000 UTC m=+1684.709936751 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/21f8e6a1-cf31-4fbf-985f-cdda8c238bbb-maas-api-tls") pod "maas-api-e2e-ait-59092fc4-79f7dcddff-l75hd" (UID: "21f8e6a1-cf31-4fbf-985f-cdda8c238bbb") : secret "maas-api-e2e-ait-59092fc4-serving-cert" not found
Jul 29 15:03:02.161949 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:02.161921    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21f8e6a1-cf31-4fbf-985f-cdda8c238bbb-openshift-service-ca\") pod \"maas-api-e2e-ait-59092fc4-79f7dcddff-l75hd\" (UID: \"21f8e6a1-cf31-4fbf-985f-cdda8c238bbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59092fc4-79f7dcddff-l75hd"
Jul 29 15:03:02.171194 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:02.171169    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-splws\" (UniqueName: \"kubernetes.io/projected/21f8e6a1-cf31-4fbf-985f-cdda8c238bbb-kube-api-access-splws\") pod \"maas-api-e2e-ait-59092fc4-79f7dcddff-l75hd\" (UID: \"21f8e6a1-cf31-4fbf-985f-cdda8c238bbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59092fc4-79f7dcddff-l75hd"
Jul 29 15:03:02.665346 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:02.665313    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21f8e6a1-cf31-4fbf-985f-cdda8c238bbb-maas-api-tls\") pod \"maas-api-e2e-ait-59092fc4-79f7dcddff-l75hd\" (UID: \"21f8e6a1-cf31-4fbf-985f-cdda8c238bbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59092fc4-79f7dcddff-l75hd"
Jul 29 15:03:02.667717 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:02.667691    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21f8e6a1-cf31-4fbf-985f-cdda8c238bbb-maas-api-tls\") pod \"maas-api-e2e-ait-59092fc4-79f7dcddff-l75hd\" (UID: \"21f8e6a1-cf31-4fbf-985f-cdda8c238bbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59092fc4-79f7dcddff-l75hd"
Jul 29 15:03:02.873424 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:02.873386    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59092fc4-79f7dcddff-l75hd"
Jul 29 15:03:03.011183 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:03.011156    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-59092fc4-79f7dcddff-l75hd"]
Jul 29 15:03:03.013350 ip-10-0-136-196 kubenswrapper[2563]: W0729 15:03:03.013325    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21f8e6a1_cf31_4fbf_985f_cdda8c238bbb.slice/crio-f06da08c645de89578faa3b623b6994d1d7a275a76e8421fdcea32f5637d71af WatchSource:0}: Error finding container f06da08c645de89578faa3b623b6994d1d7a275a76e8421fdcea32f5637d71af: Status 404 returned error can't find the container with id f06da08c645de89578faa3b623b6994d1d7a275a76e8421fdcea32f5637d71af
Jul 29 15:03:03.014777 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:03.014756    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 15:03:03.123144 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:03.123093    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59092fc4-79f7dcddff-l75hd" event={"ID":"21f8e6a1-cf31-4fbf-985f-cdda8c238bbb","Type":"ContainerStarted","Data":"f06da08c645de89578faa3b623b6994d1d7a275a76e8421fdcea32f5637d71af"}
Jul 29 15:03:04.129040 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:04.129009    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59092fc4-79f7dcddff-l75hd" event={"ID":"21f8e6a1-cf31-4fbf-985f-cdda8c238bbb","Type":"ContainerStarted","Data":"1e2e36a4d762ae2eeb9a746b73c45129f54a2a2ea2cd67dec4744bea81b63dcb"}
Jul 29 15:03:04.129360 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:04.129073    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59092fc4-79f7dcddff-l75hd"
Jul 29 15:03:04.145617 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:04.145571    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59092fc4-79f7dcddff-l75hd" podStartSLOduration=2.114023983 podStartE2EDuration="3.145557519s" totalImagesPullingTime="1.031533536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:03:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:03:04.144387012 +0000 UTC m=+1686.192844950" watchObservedRunningTime="2026-07-29 15:03:04.145557519 +0000 UTC m=+1686.194015457"
Jul 29 15:03:10.139261 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:10.139230    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59092fc4-79f7dcddff-l75hd"
Jul 29 15:03:27.522896 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:27.522859    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59092fc4-bwn9s"]
Jul 29 15:03:27.526201 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:27.526181    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59092fc4-bwn9s"
Jul 29 15:03:27.529251 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:27.529225    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mmt6p\""
Jul 29 15:03:27.536595 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:27.536562    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59092fc4-bwn9s"]
Jul 29 15:03:27.572475 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:27.572439    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f4e25a8-9b44-41a6-a9d8-2da82d1a70e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-59092fc4-bwn9s\" (UID: \"7f4e25a8-9b44-41a6-a9d8-2da82d1a70e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59092fc4-bwn9s"
Jul 29 15:03:27.673864 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:27.673818    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f4e25a8-9b44-41a6-a9d8-2da82d1a70e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-59092fc4-bwn9s\" (UID: \"7f4e25a8-9b44-41a6-a9d8-2da82d1a70e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59092fc4-bwn9s"
Jul 29 15:03:27.674528 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:27.674510    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f4e25a8-9b44-41a6-a9d8-2da82d1a70e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-59092fc4-bwn9s\" (UID: \"7f4e25a8-9b44-41a6-a9d8-2da82d1a70e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59092fc4-bwn9s"
Jul 29 15:03:27.838342 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:27.838257    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59092fc4-bwn9s"
Jul 29 15:03:27.983186 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:27.983120    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59092fc4-bwn9s"]
Jul 29 15:03:27.985920 ip-10-0-136-196 kubenswrapper[2563]: W0729 15:03:27.985889    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f4e25a8_9b44_41a6_a9d8_2da82d1a70e7.slice/crio-f449b9db1871eb101958c494dbc24af08f871740768112479104d91c2a4d1849 WatchSource:0}: Error finding container f449b9db1871eb101958c494dbc24af08f871740768112479104d91c2a4d1849: Status 404 returned error can't find the container with id f449b9db1871eb101958c494dbc24af08f871740768112479104d91c2a4d1849
Jul 29 15:03:28.246789 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:28.246746    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59092fc4-bwn9s" event={"ID":"7f4e25a8-9b44-41a6-a9d8-2da82d1a70e7","Type":"ContainerStarted","Data":"f449b9db1871eb101958c494dbc24af08f871740768112479104d91c2a4d1849"}
Jul 29 15:03:31.264502 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:31.264469    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59092fc4-bwn9s" event={"ID":"7f4e25a8-9b44-41a6-a9d8-2da82d1a70e7","Type":"ContainerStarted","Data":"540bf37ef99504ebd4bb29d422930704ff97c30475a566165fb87bddec98a90f"}
Jul 29 15:03:31.280704 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:31.280659    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59092fc4-bwn9s" podStartSLOduration=1.467465992 podStartE2EDuration="4.28064456s" totalImagesPullingTime="2.813178568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:03:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:03:31.278390724 +0000 UTC m=+1713.326848670" watchObservedRunningTime="2026-07-29 15:03:31.28064456 +0000 UTC m=+1713.329102500"
Jul 29 15:03:32.270708 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:32.270670    2563 generic.go:356] "Generic (PLEG): container finished" podID="7f4e25a8-9b44-41a6-a9d8-2da82d1a70e7" containerID="540bf37ef99504ebd4bb29d422930704ff97c30475a566165fb87bddec98a90f" exitCode=0
Jul 29 15:03:32.271078 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:32.270747    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59092fc4-bwn9s" event={"ID":"7f4e25a8-9b44-41a6-a9d8-2da82d1a70e7","Type":"ContainerDied","Data":"540bf37ef99504ebd4bb29d422930704ff97c30475a566165fb87bddec98a90f"}
Jul 29 15:03:33.404800 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:33.404775    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59092fc4-bwn9s"
Jul 29 15:03:33.424901 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:33.424869    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f4e25a8-9b44-41a6-a9d8-2da82d1a70e7-maas-api-service-ca\") pod \"7f4e25a8-9b44-41a6-a9d8-2da82d1a70e7\" (UID: \"7f4e25a8-9b44-41a6-a9d8-2da82d1a70e7\") "
Jul 29 15:03:33.425298 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:33.425272    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f4e25a8-9b44-41a6-a9d8-2da82d1a70e7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7f4e25a8-9b44-41a6-a9d8-2da82d1a70e7" (UID: "7f4e25a8-9b44-41a6-a9d8-2da82d1a70e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 15:03:33.526266 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:33.526191    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f4e25a8-9b44-41a6-a9d8-2da82d1a70e7-maas-api-service-ca\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:03:34.280749 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:34.280710    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59092fc4-bwn9s" event={"ID":"7f4e25a8-9b44-41a6-a9d8-2da82d1a70e7","Type":"ContainerDied","Data":"f449b9db1871eb101958c494dbc24af08f871740768112479104d91c2a4d1849"}
Jul 29 15:03:34.280749 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:34.280747    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59092fc4-bwn9s"
Jul 29 15:03:34.280952 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:34.280791    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f449b9db1871eb101958c494dbc24af08f871740768112479104d91c2a4d1849"
Jul 29 15:03:42.579337 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:42.579306    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-59092fc4-79f7dcddff-l75hd"]
Jul 29 15:03:42.579776 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:42.579512    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59092fc4-79f7dcddff-l75hd" podUID="21f8e6a1-cf31-4fbf-985f-cdda8c238bbb" containerName="maas-api" containerID="cri-o://1e2e36a4d762ae2eeb9a746b73c45129f54a2a2ea2cd67dec4744bea81b63dcb" gracePeriod=30
Jul 29 15:03:42.800056 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:42.800023    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59092fc4-bwn9s"]
Jul 29 15:03:42.803070 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:42.803042    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59092fc4-bwn9s"]
Jul 29 15:03:42.832314 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:42.832259    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59092fc4-79f7dcddff-l75hd"
Jul 29 15:03:42.908265 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:42.908237    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21f8e6a1-cf31-4fbf-985f-cdda8c238bbb-openshift-service-ca\") pod \"21f8e6a1-cf31-4fbf-985f-cdda8c238bbb\" (UID: \"21f8e6a1-cf31-4fbf-985f-cdda8c238bbb\") "
Jul 29 15:03:42.908444 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:42.908289    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-splws\" (UniqueName: \"kubernetes.io/projected/21f8e6a1-cf31-4fbf-985f-cdda8c238bbb-kube-api-access-splws\") pod \"21f8e6a1-cf31-4fbf-985f-cdda8c238bbb\" (UID: \"21f8e6a1-cf31-4fbf-985f-cdda8c238bbb\") "
Jul 29 15:03:42.908444 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:42.908307    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21f8e6a1-cf31-4fbf-985f-cdda8c238bbb-maas-api-tls\") pod \"21f8e6a1-cf31-4fbf-985f-cdda8c238bbb\" (UID: \"21f8e6a1-cf31-4fbf-985f-cdda8c238bbb\") "
Jul 29 15:03:42.908671 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:42.908646    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21f8e6a1-cf31-4fbf-985f-cdda8c238bbb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "21f8e6a1-cf31-4fbf-985f-cdda8c238bbb" (UID: "21f8e6a1-cf31-4fbf-985f-cdda8c238bbb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 15:03:42.914918 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:42.914892    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21f8e6a1-cf31-4fbf-985f-cdda8c238bbb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "21f8e6a1-cf31-4fbf-985f-cdda8c238bbb" (UID: "21f8e6a1-cf31-4fbf-985f-cdda8c238bbb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 15:03:42.915010 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:42.914968    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21f8e6a1-cf31-4fbf-985f-cdda8c238bbb-kube-api-access-splws" (OuterVolumeSpecName: "kube-api-access-splws") pod "21f8e6a1-cf31-4fbf-985f-cdda8c238bbb" (UID: "21f8e6a1-cf31-4fbf-985f-cdda8c238bbb"). InnerVolumeSpecName "kube-api-access-splws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 15:03:43.009362 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:43.009326    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-splws\" (UniqueName: \"kubernetes.io/projected/21f8e6a1-cf31-4fbf-985f-cdda8c238bbb-kube-api-access-splws\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:03:43.009362 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:43.009354    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21f8e6a1-cf31-4fbf-985f-cdda8c238bbb-maas-api-tls\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:03:43.009362 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:43.009364    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21f8e6a1-cf31-4fbf-985f-cdda8c238bbb-openshift-service-ca\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:03:43.321385 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:43.321342    2563 generic.go:356] "Generic (PLEG): container finished" podID="21f8e6a1-cf31-4fbf-985f-cdda8c238bbb" containerID="1e2e36a4d762ae2eeb9a746b73c45129f54a2a2ea2cd67dec4744bea81b63dcb" exitCode=0
Jul 29 15:03:43.321555 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:43.321397    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59092fc4-79f7dcddff-l75hd"
Jul 29 15:03:43.321555 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:43.321417    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59092fc4-79f7dcddff-l75hd" event={"ID":"21f8e6a1-cf31-4fbf-985f-cdda8c238bbb","Type":"ContainerDied","Data":"1e2e36a4d762ae2eeb9a746b73c45129f54a2a2ea2cd67dec4744bea81b63dcb"}
Jul 29 15:03:43.321555 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:43.321471    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59092fc4-79f7dcddff-l75hd" event={"ID":"21f8e6a1-cf31-4fbf-985f-cdda8c238bbb","Type":"ContainerDied","Data":"f06da08c645de89578faa3b623b6994d1d7a275a76e8421fdcea32f5637d71af"}
Jul 29 15:03:43.321555 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:43.321511    2563 scope.go:117] "RemoveContainer" containerID="1e2e36a4d762ae2eeb9a746b73c45129f54a2a2ea2cd67dec4744bea81b63dcb"
Jul 29 15:03:43.331446 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:43.331428    2563 scope.go:117] "RemoveContainer" containerID="1e2e36a4d762ae2eeb9a746b73c45129f54a2a2ea2cd67dec4744bea81b63dcb"
Jul 29 15:03:43.331708 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:03:43.331690    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e2e36a4d762ae2eeb9a746b73c45129f54a2a2ea2cd67dec4744bea81b63dcb\": container with ID starting with 1e2e36a4d762ae2eeb9a746b73c45129f54a2a2ea2cd67dec4744bea81b63dcb not found: ID does not exist" containerID="1e2e36a4d762ae2eeb9a746b73c45129f54a2a2ea2cd67dec4744bea81b63dcb"
Jul 29 15:03:43.331757 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:43.331717    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e2e36a4d762ae2eeb9a746b73c45129f54a2a2ea2cd67dec4744bea81b63dcb"} err="failed to get container status \"1e2e36a4d762ae2eeb9a746b73c45129f54a2a2ea2cd67dec4744bea81b63dcb\": rpc error: code = NotFound desc = could not find container \"1e2e36a4d762ae2eeb9a746b73c45129f54a2a2ea2cd67dec4744bea81b63dcb\": container with ID starting with 1e2e36a4d762ae2eeb9a746b73c45129f54a2a2ea2cd67dec4744bea81b63dcb not found: ID does not exist"
Jul 29 15:03:43.344760 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:43.344730    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-59092fc4-79f7dcddff-l75hd"]
Jul 29 15:03:43.346566 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:43.346543    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-59092fc4-79f7dcddff-l75hd"]
Jul 29 15:03:44.432775 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:44.432738    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21f8e6a1-cf31-4fbf-985f-cdda8c238bbb" path="/var/lib/kubelet/pods/21f8e6a1-cf31-4fbf-985f-cdda8c238bbb/volumes"
Jul 29 15:03:44.433247 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:44.433062    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f4e25a8-9b44-41a6-a9d8-2da82d1a70e7" path="/var/lib/kubelet/pods/7f4e25a8-9b44-41a6-a9d8-2da82d1a70e7/volumes"
Jul 29 15:03:46.370865 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:46.370833    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-079ae78c-7bd47bd4f4-xhp5d"]
Jul 29 15:03:46.371333 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:46.371246    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21f8e6a1-cf31-4fbf-985f-cdda8c238bbb" containerName="maas-api"
Jul 29 15:03:46.371333 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:46.371259    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="21f8e6a1-cf31-4fbf-985f-cdda8c238bbb" containerName="maas-api"
Jul 29 15:03:46.371333 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:46.371269    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f4e25a8-9b44-41a6-a9d8-2da82d1a70e7" containerName="revoke-keys"
Jul 29 15:03:46.371333 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:46.371275    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f4e25a8-9b44-41a6-a9d8-2da82d1a70e7" containerName="revoke-keys"
Jul 29 15:03:46.371542 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:46.371343    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f4e25a8-9b44-41a6-a9d8-2da82d1a70e7" containerName="revoke-keys"
Jul 29 15:03:46.371542 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:46.371351    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="21f8e6a1-cf31-4fbf-985f-cdda8c238bbb" containerName="maas-api"
Jul 29 15:03:46.375796 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:46.375774    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-079ae78c-7bd47bd4f4-xhp5d"
Jul 29 15:03:46.378906 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:46.378883    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-079ae78c-serving-cert\""
Jul 29 15:03:46.390987 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:46.390959    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-079ae78c-7bd47bd4f4-xhp5d"]
Jul 29 15:03:46.443458 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:46.443421    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15ba6f45-66a0-4037-ae62-bb87bfa61c90-maas-api-tls\") pod \"maas-api-e2e-migrate-079ae78c-7bd47bd4f4-xhp5d\" (UID: \"15ba6f45-66a0-4037-ae62-bb87bfa61c90\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-079ae78c-7bd47bd4f4-xhp5d"
Jul 29 15:03:46.443647 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:46.443464    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmzdk\" (UniqueName: \"kubernetes.io/projected/15ba6f45-66a0-4037-ae62-bb87bfa61c90-kube-api-access-bmzdk\") pod \"maas-api-e2e-migrate-079ae78c-7bd47bd4f4-xhp5d\" (UID: \"15ba6f45-66a0-4037-ae62-bb87bfa61c90\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-079ae78c-7bd47bd4f4-xhp5d"
Jul 29 15:03:46.443647 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:46.443531    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15ba6f45-66a0-4037-ae62-bb87bfa61c90-openshift-service-ca\") pod \"maas-api-e2e-migrate-079ae78c-7bd47bd4f4-xhp5d\" (UID: \"15ba6f45-66a0-4037-ae62-bb87bfa61c90\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-079ae78c-7bd47bd4f4-xhp5d"
Jul 29 15:03:46.544513 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:46.544475    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15ba6f45-66a0-4037-ae62-bb87bfa61c90-openshift-service-ca\") pod \"maas-api-e2e-migrate-079ae78c-7bd47bd4f4-xhp5d\" (UID: \"15ba6f45-66a0-4037-ae62-bb87bfa61c90\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-079ae78c-7bd47bd4f4-xhp5d"
Jul 29 15:03:46.544669 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:46.544574    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15ba6f45-66a0-4037-ae62-bb87bfa61c90-maas-api-tls\") pod \"maas-api-e2e-migrate-079ae78c-7bd47bd4f4-xhp5d\" (UID: \"15ba6f45-66a0-4037-ae62-bb87bfa61c90\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-079ae78c-7bd47bd4f4-xhp5d"
Jul 29 15:03:46.544669 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:46.544593    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmzdk\" (UniqueName: \"kubernetes.io/projected/15ba6f45-66a0-4037-ae62-bb87bfa61c90-kube-api-access-bmzdk\") pod \"maas-api-e2e-migrate-079ae78c-7bd47bd4f4-xhp5d\" (UID: \"15ba6f45-66a0-4037-ae62-bb87bfa61c90\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-079ae78c-7bd47bd4f4-xhp5d"
Jul 29 15:03:46.545200 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:46.545170    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15ba6f45-66a0-4037-ae62-bb87bfa61c90-openshift-service-ca\") pod \"maas-api-e2e-migrate-079ae78c-7bd47bd4f4-xhp5d\" (UID: \"15ba6f45-66a0-4037-ae62-bb87bfa61c90\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-079ae78c-7bd47bd4f4-xhp5d"
Jul 29 15:03:46.547051 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:46.547028    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15ba6f45-66a0-4037-ae62-bb87bfa61c90-maas-api-tls\") pod \"maas-api-e2e-migrate-079ae78c-7bd47bd4f4-xhp5d\" (UID: \"15ba6f45-66a0-4037-ae62-bb87bfa61c90\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-079ae78c-7bd47bd4f4-xhp5d"
Jul 29 15:03:46.558954 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:46.558931    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmzdk\" (UniqueName: \"kubernetes.io/projected/15ba6f45-66a0-4037-ae62-bb87bfa61c90-kube-api-access-bmzdk\") pod \"maas-api-e2e-migrate-079ae78c-7bd47bd4f4-xhp5d\" (UID: \"15ba6f45-66a0-4037-ae62-bb87bfa61c90\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-079ae78c-7bd47bd4f4-xhp5d"
Jul 29 15:03:46.686288 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:46.686182    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-079ae78c-7bd47bd4f4-xhp5d"
Jul 29 15:03:46.833753 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:46.833722    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-079ae78c-7bd47bd4f4-xhp5d"]
Jul 29 15:03:46.838256 ip-10-0-136-196 kubenswrapper[2563]: W0729 15:03:46.838232    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15ba6f45_66a0_4037_ae62_bb87bfa61c90.slice/crio-297662f38abdacc45c3b16a01afc2acff426654b4b57cd7cc2ee19fb79f7ccde WatchSource:0}: Error finding container 297662f38abdacc45c3b16a01afc2acff426654b4b57cd7cc2ee19fb79f7ccde: Status 404 returned error can't find the container with id 297662f38abdacc45c3b16a01afc2acff426654b4b57cd7cc2ee19fb79f7ccde
Jul 29 15:03:47.342609 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:47.342578    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-079ae78c-7bd47bd4f4-xhp5d" event={"ID":"15ba6f45-66a0-4037-ae62-bb87bfa61c90","Type":"ContainerStarted","Data":"297662f38abdacc45c3b16a01afc2acff426654b4b57cd7cc2ee19fb79f7ccde"}
Jul 29 15:03:48.349281 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:48.349253    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-079ae78c-7bd47bd4f4-xhp5d" event={"ID":"15ba6f45-66a0-4037-ae62-bb87bfa61c90","Type":"ContainerStarted","Data":"731ebaa04a27e6a048af53a751e9ca15dffd3b4301c238847a8a352eda33e58c"}
Jul 29 15:03:48.349668 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:48.349355    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-079ae78c-7bd47bd4f4-xhp5d"
Jul 29 15:03:48.375284 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:48.375237    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-079ae78c-7bd47bd4f4-xhp5d" podStartSLOduration=1.791466578 podStartE2EDuration="2.375224173s" totalImagesPullingTime="583.757595ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:03:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:03:48.370850191 +0000 UTC m=+1730.419308129" watchObservedRunningTime="2026-07-29 15:03:48.375224173 +0000 UTC m=+1730.423682111"
Jul 29 15:03:54.359714 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:03:54.359683    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-079ae78c-7bd47bd4f4-xhp5d"
Jul 29 15:04:11.362693 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:11.362658    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-079ae78c-bj7vj"]
Jul 29 15:04:11.370766 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:11.370739    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-079ae78c-bj7vj"
Jul 29 15:04:11.374631 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:11.374608    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mmt6p\""
Jul 29 15:04:11.379670 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:11.379649    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-079ae78c-bj7vj"]
Jul 29 15:04:11.460096 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:11.460066    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edd1fa96-fa87-4bef-913a-7dd4d84ab752-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-079ae78c-bj7vj\" (UID: \"edd1fa96-fa87-4bef-913a-7dd4d84ab752\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-079ae78c-bj7vj"
Jul 29 15:04:11.561079 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:11.561050    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edd1fa96-fa87-4bef-913a-7dd4d84ab752-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-079ae78c-bj7vj\" (UID: \"edd1fa96-fa87-4bef-913a-7dd4d84ab752\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-079ae78c-bj7vj"
Jul 29 15:04:11.561664 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:11.561644    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edd1fa96-fa87-4bef-913a-7dd4d84ab752-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-079ae78c-bj7vj\" (UID: \"edd1fa96-fa87-4bef-913a-7dd4d84ab752\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-079ae78c-bj7vj"
Jul 29 15:04:11.683164 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:11.683065    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-079ae78c-bj7vj"
Jul 29 15:04:11.817703 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:11.817659    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-079ae78c-bj7vj"]
Jul 29 15:04:11.821780 ip-10-0-136-196 kubenswrapper[2563]: W0729 15:04:11.821753    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedd1fa96_fa87_4bef_913a_7dd4d84ab752.slice/crio-7b844890b4fbc7be076189a61b34cffbd84f00e563a48394ade4d2cb01908460 WatchSource:0}: Error finding container 7b844890b4fbc7be076189a61b34cffbd84f00e563a48394ade4d2cb01908460: Status 404 returned error can't find the container with id 7b844890b4fbc7be076189a61b34cffbd84f00e563a48394ade4d2cb01908460
Jul 29 15:04:12.464270 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:12.464233    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-079ae78c-bj7vj" event={"ID":"edd1fa96-fa87-4bef-913a-7dd4d84ab752","Type":"ContainerStarted","Data":"b39a25005a7e8eeecaa2906e9a76a6197d47bf93480a439b1c33ad5a50008ef1"}
Jul 29 15:04:12.464686 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:12.464285    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-079ae78c-bj7vj" event={"ID":"edd1fa96-fa87-4bef-913a-7dd4d84ab752","Type":"ContainerStarted","Data":"7b844890b4fbc7be076189a61b34cffbd84f00e563a48394ade4d2cb01908460"}
Jul 29 15:04:12.479842 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:12.479798    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-079ae78c-bj7vj" podStartSLOduration=1.479783399 podStartE2EDuration="1.479783399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:04:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:04:12.478483625 +0000 UTC m=+1754.526941565" watchObservedRunningTime="2026-07-29 15:04:12.479783399 +0000 UTC m=+1754.528241337"
Jul 29 15:04:13.469915 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:13.469882    2563 generic.go:356] "Generic (PLEG): container finished" podID="edd1fa96-fa87-4bef-913a-7dd4d84ab752" containerID="b39a25005a7e8eeecaa2906e9a76a6197d47bf93480a439b1c33ad5a50008ef1" exitCode=0
Jul 29 15:04:13.470295 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:13.469975    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-079ae78c-bj7vj" event={"ID":"edd1fa96-fa87-4bef-913a-7dd4d84ab752","Type":"ContainerDied","Data":"b39a25005a7e8eeecaa2906e9a76a6197d47bf93480a439b1c33ad5a50008ef1"}
Jul 29 15:04:14.601788 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:14.601761    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-079ae78c-bj7vj"
Jul 29 15:04:14.684748 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:14.684720    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edd1fa96-fa87-4bef-913a-7dd4d84ab752-maas-api-service-ca\") pod \"edd1fa96-fa87-4bef-913a-7dd4d84ab752\" (UID: \"edd1fa96-fa87-4bef-913a-7dd4d84ab752\") "
Jul 29 15:04:14.685041 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:14.685021    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edd1fa96-fa87-4bef-913a-7dd4d84ab752-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "edd1fa96-fa87-4bef-913a-7dd4d84ab752" (UID: "edd1fa96-fa87-4bef-913a-7dd4d84ab752"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 15:04:14.785826 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:14.785764    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edd1fa96-fa87-4bef-913a-7dd4d84ab752-maas-api-service-ca\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:04:15.483043 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:15.483004    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-079ae78c-bj7vj" event={"ID":"edd1fa96-fa87-4bef-913a-7dd4d84ab752","Type":"ContainerDied","Data":"7b844890b4fbc7be076189a61b34cffbd84f00e563a48394ade4d2cb01908460"}
Jul 29 15:04:15.483237 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:15.483060    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b844890b4fbc7be076189a61b34cffbd84f00e563a48394ade4d2cb01908460"
Jul 29 15:04:15.483237 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:15.483022    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-079ae78c-bj7vj"
Jul 29 15:04:17.188456 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:17.188425    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-079ae78c-7bd47bd4f4-xhp5d"]
Jul 29 15:04:17.188918 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:17.188642    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-079ae78c-7bd47bd4f4-xhp5d" podUID="15ba6f45-66a0-4037-ae62-bb87bfa61c90" containerName="maas-api" containerID="cri-o://731ebaa04a27e6a048af53a751e9ca15dffd3b4301c238847a8a352eda33e58c" gracePeriod=30
Jul 29 15:04:17.420384 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:17.420349    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-079ae78c-bj7vj"]
Jul 29 15:04:17.420618 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:17.420605    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-079ae78c-bj7vj"]
Jul 29 15:04:17.449799 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:17.449746    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-079ae78c-7bd47bd4f4-xhp5d"
Jul 29 15:04:17.494887 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:17.494857    2563 generic.go:356] "Generic (PLEG): container finished" podID="15ba6f45-66a0-4037-ae62-bb87bfa61c90" containerID="731ebaa04a27e6a048af53a751e9ca15dffd3b4301c238847a8a352eda33e58c" exitCode=0
Jul 29 15:04:17.495044 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:17.494923    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-079ae78c-7bd47bd4f4-xhp5d"
Jul 29 15:04:17.495044 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:17.494943    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-079ae78c-7bd47bd4f4-xhp5d" event={"ID":"15ba6f45-66a0-4037-ae62-bb87bfa61c90","Type":"ContainerDied","Data":"731ebaa04a27e6a048af53a751e9ca15dffd3b4301c238847a8a352eda33e58c"}
Jul 29 15:04:17.495044 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:17.495007    2563 scope.go:117] "RemoveContainer" containerID="731ebaa04a27e6a048af53a751e9ca15dffd3b4301c238847a8a352eda33e58c"
Jul 29 15:04:17.498424 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:17.494996    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-079ae78c-7bd47bd4f4-xhp5d" event={"ID":"15ba6f45-66a0-4037-ae62-bb87bfa61c90","Type":"ContainerDied","Data":"297662f38abdacc45c3b16a01afc2acff426654b4b57cd7cc2ee19fb79f7ccde"}
Jul 29 15:04:17.509645 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:17.509627    2563 scope.go:117] "RemoveContainer" containerID="731ebaa04a27e6a048af53a751e9ca15dffd3b4301c238847a8a352eda33e58c"
Jul 29 15:04:17.509925 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:04:17.509909    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"731ebaa04a27e6a048af53a751e9ca15dffd3b4301c238847a8a352eda33e58c\": container with ID starting with 731ebaa04a27e6a048af53a751e9ca15dffd3b4301c238847a8a352eda33e58c not found: ID does not exist" containerID="731ebaa04a27e6a048af53a751e9ca15dffd3b4301c238847a8a352eda33e58c"
Jul 29 15:04:17.509968 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:17.509932    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"731ebaa04a27e6a048af53a751e9ca15dffd3b4301c238847a8a352eda33e58c"} err="failed to get container status \"731ebaa04a27e6a048af53a751e9ca15dffd3b4301c238847a8a352eda33e58c\": rpc error: code = NotFound desc = could not find container \"731ebaa04a27e6a048af53a751e9ca15dffd3b4301c238847a8a352eda33e58c\": container with ID starting with 731ebaa04a27e6a048af53a751e9ca15dffd3b4301c238847a8a352eda33e58c not found: ID does not exist"
Jul 29 15:04:17.615128 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:17.615098    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15ba6f45-66a0-4037-ae62-bb87bfa61c90-openshift-service-ca\") pod \"15ba6f45-66a0-4037-ae62-bb87bfa61c90\" (UID: \"15ba6f45-66a0-4037-ae62-bb87bfa61c90\") "
Jul 29 15:04:17.615303 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:17.615193    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15ba6f45-66a0-4037-ae62-bb87bfa61c90-maas-api-tls\") pod \"15ba6f45-66a0-4037-ae62-bb87bfa61c90\" (UID: \"15ba6f45-66a0-4037-ae62-bb87bfa61c90\") "
Jul 29 15:04:17.615303 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:17.615227    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmzdk\" (UniqueName: \"kubernetes.io/projected/15ba6f45-66a0-4037-ae62-bb87bfa61c90-kube-api-access-bmzdk\") pod \"15ba6f45-66a0-4037-ae62-bb87bfa61c90\" (UID: \"15ba6f45-66a0-4037-ae62-bb87bfa61c90\") "
Jul 29 15:04:17.615472 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:17.615448    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15ba6f45-66a0-4037-ae62-bb87bfa61c90-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "15ba6f45-66a0-4037-ae62-bb87bfa61c90" (UID: "15ba6f45-66a0-4037-ae62-bb87bfa61c90"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 15:04:17.617320 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:17.617293    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15ba6f45-66a0-4037-ae62-bb87bfa61c90-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "15ba6f45-66a0-4037-ae62-bb87bfa61c90" (UID: "15ba6f45-66a0-4037-ae62-bb87bfa61c90"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 15:04:17.617320 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:17.617303    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15ba6f45-66a0-4037-ae62-bb87bfa61c90-kube-api-access-bmzdk" (OuterVolumeSpecName: "kube-api-access-bmzdk") pod "15ba6f45-66a0-4037-ae62-bb87bfa61c90" (UID: "15ba6f45-66a0-4037-ae62-bb87bfa61c90"). InnerVolumeSpecName "kube-api-access-bmzdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 15:04:17.716149 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:17.716107    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15ba6f45-66a0-4037-ae62-bb87bfa61c90-openshift-service-ca\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:04:17.716296 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:17.716164    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15ba6f45-66a0-4037-ae62-bb87bfa61c90-maas-api-tls\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:04:17.716296 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:17.716175    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmzdk\" (UniqueName: \"kubernetes.io/projected/15ba6f45-66a0-4037-ae62-bb87bfa61c90-kube-api-access-bmzdk\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:04:17.816215 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:17.816186    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-079ae78c-7bd47bd4f4-xhp5d"]
Jul 29 15:04:17.821038 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:17.821017    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-079ae78c-7bd47bd4f4-xhp5d"]
Jul 29 15:04:18.432605 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:18.432568    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15ba6f45-66a0-4037-ae62-bb87bfa61c90" path="/var/lib/kubelet/pods/15ba6f45-66a0-4037-ae62-bb87bfa61c90/volumes"
Jul 29 15:04:18.433082 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:18.433012    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edd1fa96-fa87-4bef-913a-7dd4d84ab752" path="/var/lib/kubelet/pods/edd1fa96-fa87-4bef-913a-7dd4d84ab752/volumes"
Jul 29 15:04:20.335538 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:20.335499    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-43963a57-6c75bb746c-lwpwn"]
Jul 29 15:04:20.336036 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:20.336015    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edd1fa96-fa87-4bef-913a-7dd4d84ab752" containerName="revoke-keys"
Jul 29 15:04:20.336108 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:20.336040    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="edd1fa96-fa87-4bef-913a-7dd4d84ab752" containerName="revoke-keys"
Jul 29 15:04:20.336108 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:20.336062    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15ba6f45-66a0-4037-ae62-bb87bfa61c90" containerName="maas-api"
Jul 29 15:04:20.336108 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:20.336072    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="15ba6f45-66a0-4037-ae62-bb87bfa61c90" containerName="maas-api"
Jul 29 15:04:20.336290 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:20.336200    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="15ba6f45-66a0-4037-ae62-bb87bfa61c90" containerName="maas-api"
Jul 29 15:04:20.336290 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:20.336222    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="edd1fa96-fa87-4bef-913a-7dd4d84ab752" containerName="revoke-keys"
Jul 29 15:04:20.340456 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:20.340435    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-43963a57-6c75bb746c-lwpwn"
Jul 29 15:04:20.342481 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:20.342458    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-43963a57-serving-cert\""
Jul 29 15:04:20.349409 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:20.349387    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-43963a57-6c75bb746c-lwpwn"]
Jul 29 15:04:20.440071 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:20.440043    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef-maas-api-tls\") pod \"maas-api-e2e-ait-43963a57-6c75bb746c-lwpwn\" (UID: \"3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-43963a57-6c75bb746c-lwpwn"
Jul 29 15:04:20.440071 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:20.440074    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts75q\" (UniqueName: \"kubernetes.io/projected/3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef-kube-api-access-ts75q\") pod \"maas-api-e2e-ait-43963a57-6c75bb746c-lwpwn\" (UID: \"3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-43963a57-6c75bb746c-lwpwn"
Jul 29 15:04:20.440287 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:20.440159    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef-openshift-service-ca\") pod \"maas-api-e2e-ait-43963a57-6c75bb746c-lwpwn\" (UID: \"3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-43963a57-6c75bb746c-lwpwn"
Jul 29 15:04:20.541236 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:20.541194    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef-openshift-service-ca\") pod \"maas-api-e2e-ait-43963a57-6c75bb746c-lwpwn\" (UID: \"3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-43963a57-6c75bb746c-lwpwn"
Jul 29 15:04:20.541402 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:20.541277    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef-maas-api-tls\") pod \"maas-api-e2e-ait-43963a57-6c75bb746c-lwpwn\" (UID: \"3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-43963a57-6c75bb746c-lwpwn"
Jul 29 15:04:20.541402 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:20.541299    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts75q\" (UniqueName: \"kubernetes.io/projected/3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef-kube-api-access-ts75q\") pod \"maas-api-e2e-ait-43963a57-6c75bb746c-lwpwn\" (UID: \"3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-43963a57-6c75bb746c-lwpwn"
Jul 29 15:04:20.541483 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:04:20.541417    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-43963a57-serving-cert: secret "maas-api-e2e-ait-43963a57-serving-cert" not found
Jul 29 15:04:20.541522 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:04:20.541492    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef-maas-api-tls podName:3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef nodeName:}" failed. No retries permitted until 2026-07-29 15:04:21.04147284 +0000 UTC m=+1763.089930770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef-maas-api-tls") pod "maas-api-e2e-ait-43963a57-6c75bb746c-lwpwn" (UID: "3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef") : secret "maas-api-e2e-ait-43963a57-serving-cert" not found
Jul 29 15:04:20.541889 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:20.541869    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef-openshift-service-ca\") pod \"maas-api-e2e-ait-43963a57-6c75bb746c-lwpwn\" (UID: \"3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-43963a57-6c75bb746c-lwpwn"
Jul 29 15:04:20.548540 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:20.548516    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts75q\" (UniqueName: \"kubernetes.io/projected/3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef-kube-api-access-ts75q\") pod \"maas-api-e2e-ait-43963a57-6c75bb746c-lwpwn\" (UID: \"3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-43963a57-6c75bb746c-lwpwn"
Jul 29 15:04:21.045342 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:21.045307    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef-maas-api-tls\") pod \"maas-api-e2e-ait-43963a57-6c75bb746c-lwpwn\" (UID: \"3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-43963a57-6c75bb746c-lwpwn"
Jul 29 15:04:21.047694 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:21.047667    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef-maas-api-tls\") pod \"maas-api-e2e-ait-43963a57-6c75bb746c-lwpwn\" (UID: \"3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-43963a57-6c75bb746c-lwpwn"
Jul 29 15:04:21.252223 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:21.252184    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-43963a57-6c75bb746c-lwpwn"
Jul 29 15:04:21.598731 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:21.598665    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-43963a57-6c75bb746c-lwpwn"]
Jul 29 15:04:21.601393 ip-10-0-136-196 kubenswrapper[2563]: W0729 15:04:21.601365    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e14e8c3_4e37_4125_83f0_8ee9bc2bc2ef.slice/crio-387eb5161af0e1601ac4430664996cecc447742ed9a7affc453cd11eab476d5c WatchSource:0}: Error finding container 387eb5161af0e1601ac4430664996cecc447742ed9a7affc453cd11eab476d5c: Status 404 returned error can't find the container with id 387eb5161af0e1601ac4430664996cecc447742ed9a7affc453cd11eab476d5c
Jul 29 15:04:22.521649 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:22.521561    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-43963a57-6c75bb746c-lwpwn" event={"ID":"3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef","Type":"ContainerStarted","Data":"4a7e2c30e51ac6c5085907794e9da230e68596ec6cbf7fea50dffcedf12d429c"}
Jul 29 15:04:22.521649 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:22.521625    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-43963a57-6c75bb746c-lwpwn" event={"ID":"3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef","Type":"ContainerStarted","Data":"387eb5161af0e1601ac4430664996cecc447742ed9a7affc453cd11eab476d5c"}
Jul 29 15:04:22.521830 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:22.521774    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-43963a57-6c75bb746c-lwpwn"
Jul 29 15:04:22.539036 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:22.538985    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-43963a57-6c75bb746c-lwpwn" podStartSLOduration=1.918267443 podStartE2EDuration="2.538969044s" totalImagesPullingTime="620.701601ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:04:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:04:22.537455096 +0000 UTC m=+1764.585913034" watchObservedRunningTime="2026-07-29 15:04:22.538969044 +0000 UTC m=+1764.587427025"
Jul 29 15:04:28.532253 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:28.532224    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-43963a57-6c75bb746c-lwpwn"
Jul 29 15:04:46.147717 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:46.147678    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-43963a57-n5k6h"]
Jul 29 15:04:46.151504 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:46.151483    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-43963a57-n5k6h"
Jul 29 15:04:46.154960 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:46.154929    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mmt6p\""
Jul 29 15:04:46.164435 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:46.164390    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-43963a57-n5k6h"]
Jul 29 15:04:46.266588 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:46.266552    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dab58e1a-aefc-416a-a452-60d677736f39-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-43963a57-n5k6h\" (UID: \"dab58e1a-aefc-416a-a452-60d677736f39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-43963a57-n5k6h"
Jul 29 15:04:46.367827 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:46.367789    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dab58e1a-aefc-416a-a452-60d677736f39-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-43963a57-n5k6h\" (UID: \"dab58e1a-aefc-416a-a452-60d677736f39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-43963a57-n5k6h"
Jul 29 15:04:46.368498 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:46.368470    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dab58e1a-aefc-416a-a452-60d677736f39-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-43963a57-n5k6h\" (UID: \"dab58e1a-aefc-416a-a452-60d677736f39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-43963a57-n5k6h"
Jul 29 15:04:46.464716 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:46.464685    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-43963a57-n5k6h"
Jul 29 15:04:46.622172 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:46.622038    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-43963a57-n5k6h"]
Jul 29 15:04:46.624843 ip-10-0-136-196 kubenswrapper[2563]: W0729 15:04:46.624815    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddab58e1a_aefc_416a_a452_60d677736f39.slice/crio-b3728e544acd4500e155d3f46ece2a0d25b9ec75d1470a60a2a15d5b79b6b8ad WatchSource:0}: Error finding container b3728e544acd4500e155d3f46ece2a0d25b9ec75d1470a60a2a15d5b79b6b8ad: Status 404 returned error can't find the container with id b3728e544acd4500e155d3f46ece2a0d25b9ec75d1470a60a2a15d5b79b6b8ad
Jul 29 15:04:46.643248 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:46.643214    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-43963a57-n5k6h" event={"ID":"dab58e1a-aefc-416a-a452-60d677736f39","Type":"ContainerStarted","Data":"b3728e544acd4500e155d3f46ece2a0d25b9ec75d1470a60a2a15d5b79b6b8ad"}
Jul 29 15:04:47.649452 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:47.649364    2563 generic.go:356] "Generic (PLEG): container finished" podID="dab58e1a-aefc-416a-a452-60d677736f39" containerID="3bf765168f0e3c4839649b638e487865ea1b6bc3d0daea22f426b51d5cb1c3e2" exitCode=0
Jul 29 15:04:47.649452 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:47.649427    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-43963a57-n5k6h" event={"ID":"dab58e1a-aefc-416a-a452-60d677736f39","Type":"ContainerDied","Data":"3bf765168f0e3c4839649b638e487865ea1b6bc3d0daea22f426b51d5cb1c3e2"}
Jul 29 15:04:48.780015 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:48.779993    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-43963a57-n5k6h"
Jul 29 15:04:48.890562 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:48.890535    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dab58e1a-aefc-416a-a452-60d677736f39-maas-api-service-ca\") pod \"dab58e1a-aefc-416a-a452-60d677736f39\" (UID: \"dab58e1a-aefc-416a-a452-60d677736f39\") "
Jul 29 15:04:48.890861 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:48.890841    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dab58e1a-aefc-416a-a452-60d677736f39-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dab58e1a-aefc-416a-a452-60d677736f39" (UID: "dab58e1a-aefc-416a-a452-60d677736f39"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 15:04:48.991924 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:48.991898    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dab58e1a-aefc-416a-a452-60d677736f39-maas-api-service-ca\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:04:49.663380 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:49.663353    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-43963a57-n5k6h"
Jul 29 15:04:49.663380 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:49.663369    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-43963a57-n5k6h" event={"ID":"dab58e1a-aefc-416a-a452-60d677736f39","Type":"ContainerDied","Data":"b3728e544acd4500e155d3f46ece2a0d25b9ec75d1470a60a2a15d5b79b6b8ad"}
Jul 29 15:04:49.663641 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:49.663415    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3728e544acd4500e155d3f46ece2a0d25b9ec75d1470a60a2a15d5b79b6b8ad"
Jul 29 15:04:51.065001 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:51.064931    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-43963a57-6c75bb746c-lwpwn"]
Jul 29 15:04:51.065437 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:51.065116    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-43963a57-6c75bb746c-lwpwn" podUID="3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef" containerName="maas-api" containerID="cri-o://4a7e2c30e51ac6c5085907794e9da230e68596ec6cbf7fea50dffcedf12d429c" gracePeriod=30
Jul 29 15:04:51.335803 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:51.335781    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-43963a57-6c75bb746c-lwpwn"
Jul 29 15:04:51.414022 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:51.413989    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ts75q\" (UniqueName: \"kubernetes.io/projected/3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef-kube-api-access-ts75q\") pod \"3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef\" (UID: \"3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef\") "
Jul 29 15:04:51.414247 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:51.414086    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef-maas-api-tls\") pod \"3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef\" (UID: \"3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef\") "
Jul 29 15:04:51.414247 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:51.414178    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef-openshift-service-ca\") pod \"3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef\" (UID: \"3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef\") "
Jul 29 15:04:51.414720 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:51.414694    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef" (UID: "3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 15:04:51.416243 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:51.416218    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef-kube-api-access-ts75q" (OuterVolumeSpecName: "kube-api-access-ts75q") pod "3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef" (UID: "3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef"). InnerVolumeSpecName "kube-api-access-ts75q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 15:04:51.416340 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:51.416256    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef" (UID: "3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 15:04:51.495954 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:51.495915    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-43963a57-n5k6h"]
Jul 29 15:04:51.506108 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:51.506075    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-43963a57-n5k6h"]
Jul 29 15:04:51.515791 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:51.515765    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef-maas-api-tls\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:04:51.515915 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:51.515795    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef-openshift-service-ca\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:04:51.515915 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:51.515813    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts75q\" (UniqueName: \"kubernetes.io/projected/3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef-kube-api-access-ts75q\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:04:51.673553 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:51.673456    2563 generic.go:356] "Generic (PLEG): container finished" podID="3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef" containerID="4a7e2c30e51ac6c5085907794e9da230e68596ec6cbf7fea50dffcedf12d429c" exitCode=0
Jul 29 15:04:51.673553 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:51.673502    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-43963a57-6c75bb746c-lwpwn"
Jul 29 15:04:51.673553 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:51.673519    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-43963a57-6c75bb746c-lwpwn" event={"ID":"3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef","Type":"ContainerDied","Data":"4a7e2c30e51ac6c5085907794e9da230e68596ec6cbf7fea50dffcedf12d429c"}
Jul 29 15:04:51.673802 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:51.673575    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-43963a57-6c75bb746c-lwpwn" event={"ID":"3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef","Type":"ContainerDied","Data":"387eb5161af0e1601ac4430664996cecc447742ed9a7affc453cd11eab476d5c"}
Jul 29 15:04:51.673802 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:51.673605    2563 scope.go:117] "RemoveContainer" containerID="4a7e2c30e51ac6c5085907794e9da230e68596ec6cbf7fea50dffcedf12d429c"
Jul 29 15:04:51.682696 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:51.682681    2563 scope.go:117] "RemoveContainer" containerID="4a7e2c30e51ac6c5085907794e9da230e68596ec6cbf7fea50dffcedf12d429c"
Jul 29 15:04:51.682948 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:04:51.682930    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a7e2c30e51ac6c5085907794e9da230e68596ec6cbf7fea50dffcedf12d429c\": container with ID starting with 4a7e2c30e51ac6c5085907794e9da230e68596ec6cbf7fea50dffcedf12d429c not found: ID does not exist" containerID="4a7e2c30e51ac6c5085907794e9da230e68596ec6cbf7fea50dffcedf12d429c"
Jul 29 15:04:51.682996 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:51.682956    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a7e2c30e51ac6c5085907794e9da230e68596ec6cbf7fea50dffcedf12d429c"} err="failed to get container status \"4a7e2c30e51ac6c5085907794e9da230e68596ec6cbf7fea50dffcedf12d429c\": rpc error: code = NotFound desc = could not find container \"4a7e2c30e51ac6c5085907794e9da230e68596ec6cbf7fea50dffcedf12d429c\": container with ID starting with 4a7e2c30e51ac6c5085907794e9da230e68596ec6cbf7fea50dffcedf12d429c not found: ID does not exist"
Jul 29 15:04:51.695600 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:51.695578    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-43963a57-6c75bb746c-lwpwn"]
Jul 29 15:04:51.699547 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:51.699528    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-43963a57-6c75bb746c-lwpwn"]
Jul 29 15:04:52.432398 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:52.432363    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef" path="/var/lib/kubelet/pods/3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef/volumes"
Jul 29 15:04:52.432766 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:52.432674    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dab58e1a-aefc-416a-a452-60d677736f39" path="/var/lib/kubelet/pods/dab58e1a-aefc-416a-a452-60d677736f39/volumes"
Jul 29 15:04:53.525524 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:53.525485    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"]
Jul 29 15:04:53.526009 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:53.525987    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dab58e1a-aefc-416a-a452-60d677736f39" containerName="revoke-keys"
Jul 29 15:04:53.526117 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:53.526011    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dab58e1a-aefc-416a-a452-60d677736f39" containerName="revoke-keys"
Jul 29 15:04:53.526117 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:53.526033    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef" containerName="maas-api"
Jul 29 15:04:53.526117 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:53.526040    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef" containerName="maas-api"
Jul 29 15:04:53.526263 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:53.526123    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="dab58e1a-aefc-416a-a452-60d677736f39" containerName="revoke-keys"
Jul 29 15:04:53.526263 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:53.526151    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e14e8c3-4e37-4125-83f0-8ee9bc2bc2ef" containerName="maas-api"
Jul 29 15:04:53.529498 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:53.529480    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"
Jul 29 15:04:53.531977 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:53.531953    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 29 15:04:53.532066 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:53.531957    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-19551b81-openshift-default-dockercfg-8bnm7\""
Jul 29 15:04:53.548269 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:53.548243    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"]
Jul 29 15:04:53.634014 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:53.633974    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd90a641-bffb-4137-aacb-4d8848864bb8-credential-socket\") pod \"e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq\" (UID: \"fd90a641-bffb-4137-aacb-4d8848864bb8\") " pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"
Jul 29 15:04:53.634014 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:53.634018    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd90a641-bffb-4137-aacb-4d8848864bb8-istio-envoy\") pod \"e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq\" (UID: \"fd90a641-bffb-4137-aacb-4d8848864bb8\") " pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"
Jul 29 15:04:53.634254 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:53.634086    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd90a641-bffb-4137-aacb-4d8848864bb8-istio-token\") pod \"e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq\" (UID: \"fd90a641-bffb-4137-aacb-4d8848864bb8\") " pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"
Jul 29 15:04:53.634254 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:53.634148    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd90a641-bffb-4137-aacb-4d8848864bb8-istio-podinfo\") pod \"e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq\" (UID: \"fd90a641-bffb-4137-aacb-4d8848864bb8\") " pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"
Jul 29 15:04:53.634254 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:53.634179    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd90a641-bffb-4137-aacb-4d8848864bb8-workload-certs\") pod \"e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq\" (UID: \"fd90a641-bffb-4137-aacb-4d8848864bb8\") " pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"
Jul 29 15:04:53.634356 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:53.634251    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd90a641-bffb-4137-aacb-4d8848864bb8-istio-data\") pod \"e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq\" (UID: \"fd90a641-bffb-4137-aacb-4d8848864bb8\") " pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"
Jul 29 15:04:53.634356 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:53.634278    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd90a641-bffb-4137-aacb-4d8848864bb8-istiod-ca-cert\") pod \"e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq\" (UID: \"fd90a641-bffb-4137-aacb-4d8848864bb8\") " pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"
Jul 29 15:04:53.634356 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:53.634297    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9lhx\" (UniqueName: \"kubernetes.io/projected/fd90a641-bffb-4137-aacb-4d8848864bb8-kube-api-access-r9lhx\") pod \"e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq\" (UID: \"fd90a641-bffb-4137-aacb-4d8848864bb8\") " pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"
Jul 29 15:04:53.634356 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:53.634348    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd90a641-bffb-4137-aacb-4d8848864bb8-workload-socket\") pod \"e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq\" (UID: \"fd90a641-bffb-4137-aacb-4d8848864bb8\") " pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"
Jul 29 15:04:53.735022 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:53.734989    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd90a641-bffb-4137-aacb-4d8848864bb8-istio-podinfo\") pod \"e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq\" (UID: \"fd90a641-bffb-4137-aacb-4d8848864bb8\") " pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"
Jul 29 15:04:53.735022 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:53.735027    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd90a641-bffb-4137-aacb-4d8848864bb8-workload-certs\") pod \"e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq\" (UID: \"fd90a641-bffb-4137-aacb-4d8848864bb8\") " pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"
Jul 29 15:04:53.735268 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:53.735053    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd90a641-bffb-4137-aacb-4d8848864bb8-istio-data\") pod \"e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq\" (UID: \"fd90a641-bffb-4137-aacb-4d8848864bb8\") " pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"
Jul 29 15:04:53.735268 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:53.735078    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd90a641-bffb-4137-aacb-4d8848864bb8-istiod-ca-cert\") pod \"e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq\" (UID: \"fd90a641-bffb-4137-aacb-4d8848864bb8\") " pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"
Jul 29 15:04:53.735268 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:53.735104    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9lhx\" (UniqueName: \"kubernetes.io/projected/fd90a641-bffb-4137-aacb-4d8848864bb8-kube-api-access-r9lhx\") pod \"e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq\" (UID: \"fd90a641-bffb-4137-aacb-4d8848864bb8\") " pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"
Jul 29 15:04:53.735268 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:53.735189    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd90a641-bffb-4137-aacb-4d8848864bb8-workload-socket\") pod \"e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq\" (UID: \"fd90a641-bffb-4137-aacb-4d8848864bb8\") " pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"
Jul 29 15:04:53.735268 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:53.735254    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd90a641-bffb-4137-aacb-4d8848864bb8-credential-socket\") pod \"e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq\" (UID: \"fd90a641-bffb-4137-aacb-4d8848864bb8\") " pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"
Jul 29 15:04:53.735530 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:53.735288    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd90a641-bffb-4137-aacb-4d8848864bb8-istio-envoy\") pod \"e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq\" (UID: \"fd90a641-bffb-4137-aacb-4d8848864bb8\") " pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"
Jul 29 15:04:53.735530 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:53.735347    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd90a641-bffb-4137-aacb-4d8848864bb8-istio-token\") pod \"e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq\" (UID: \"fd90a641-bffb-4137-aacb-4d8848864bb8\") " pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"
Jul 29 15:04:53.735645 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:53.735536    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd90a641-bffb-4137-aacb-4d8848864bb8-workload-certs\") pod \"e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq\" (UID: \"fd90a641-bffb-4137-aacb-4d8848864bb8\") " pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"
Jul 29 15:04:53.735645 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:53.735540    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd90a641-bffb-4137-aacb-4d8848864bb8-istio-data\") pod \"e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq\" (UID: \"fd90a641-bffb-4137-aacb-4d8848864bb8\") " pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"
Jul 29 15:04:53.735776 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:53.735752    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd90a641-bffb-4137-aacb-4d8848864bb8-credential-socket\") pod \"e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq\" (UID: \"fd90a641-bffb-4137-aacb-4d8848864bb8\") " pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"
Jul 29 15:04:53.735836 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:53.735759    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd90a641-bffb-4137-aacb-4d8848864bb8-workload-socket\") pod \"e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq\" (UID: \"fd90a641-bffb-4137-aacb-4d8848864bb8\") " pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"
Jul 29 15:04:53.736035 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:53.736013    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd90a641-bffb-4137-aacb-4d8848864bb8-istiod-ca-cert\") pod \"e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq\" (UID: \"fd90a641-bffb-4137-aacb-4d8848864bb8\") " pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"
Jul 29 15:04:53.737506 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:53.737478    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd90a641-bffb-4137-aacb-4d8848864bb8-istio-envoy\") pod \"e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq\" (UID: \"fd90a641-bffb-4137-aacb-4d8848864bb8\") " pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"
Jul 29 15:04:53.737614 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:53.737581    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd90a641-bffb-4137-aacb-4d8848864bb8-istio-podinfo\") pod \"e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq\" (UID: \"fd90a641-bffb-4137-aacb-4d8848864bb8\") " pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"
Jul 29 15:04:53.746867 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:53.746837    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9lhx\" (UniqueName: \"kubernetes.io/projected/fd90a641-bffb-4137-aacb-4d8848864bb8-kube-api-access-r9lhx\") pod \"e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq\" (UID: \"fd90a641-bffb-4137-aacb-4d8848864bb8\") " pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"
Jul 29 15:04:53.747514 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:53.747497    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd90a641-bffb-4137-aacb-4d8848864bb8-istio-token\") pod \"e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq\" (UID: \"fd90a641-bffb-4137-aacb-4d8848864bb8\") " pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"
Jul 29 15:04:53.839790 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:53.839710    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"
Jul 29 15:04:53.982870 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:53.982834    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"]
Jul 29 15:04:53.985646 ip-10-0-136-196 kubenswrapper[2563]: W0729 15:04:53.985616    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd90a641_bffb_4137_aacb_4d8848864bb8.slice/crio-49c77db8684d703d05cc5d616a51f856b5bd4e84b98b52464b44e6536faa488b WatchSource:0}: Error finding container 49c77db8684d703d05cc5d616a51f856b5bd4e84b98b52464b44e6536faa488b: Status 404 returned error can't find the container with id 49c77db8684d703d05cc5d616a51f856b5bd4e84b98b52464b44e6536faa488b
Jul 29 15:04:54.689199 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:54.689161    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq" event={"ID":"fd90a641-bffb-4137-aacb-4d8848864bb8","Type":"ContainerStarted","Data":"49c77db8684d703d05cc5d616a51f856b5bd4e84b98b52464b44e6536faa488b"}
Jul 29 15:04:58.779028 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:58.779005    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xnm95_23d79588-937d-4c6d-8760-a081024f608e/console-operator/1.log"
Jul 29 15:04:58.779440 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:58.779213    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xnm95_23d79588-937d-4c6d-8760-a081024f608e/console-operator/1.log"
Jul 29 15:04:58.817395 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:58.817362    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:04:58.817475 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:58.817428    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:04:58.817475 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:58.817455    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:04:59.714842 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:59.714801    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq" event={"ID":"fd90a641-bffb-4137-aacb-4d8848864bb8","Type":"ContainerStarted","Data":"26380c30eb9f7acbcd692d6ba7ef93f8399e9b798a5dbba9db714e466547c0c5"}
Jul 29 15:04:59.735711 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:59.735664    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq" podStartSLOduration=1.9062088149999998 podStartE2EDuration="6.735648118s" totalImagesPullingTime="4.829439303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:04:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:04:59.733341685 +0000 UTC m=+1801.781799626" watchObservedRunningTime="2026-07-29 15:04:59.735648118 +0000 UTC m=+1801.784106126"
Jul 29 15:04:59.840831 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:59.840793    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"
Jul 29 15:04:59.849246 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:04:59.849219    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"
Jul 29 15:05:00.719459 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:00.719426    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"
Jul 29 15:05:00.720362 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:00.720340    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"
Jul 29 15:05:01.198705 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:01.198614    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-19551b81-d9cf7f57d-qvmlr"]
Jul 29 15:05:01.202254 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:01.202227    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-19551b81-d9cf7f57d-qvmlr"
Jul 29 15:05:01.204846 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:01.204826    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-19551b81-serving-cert\""
Jul 29 15:05:01.217239 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:01.217212    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-19551b81-d9cf7f57d-qvmlr"]
Jul 29 15:05:01.303715 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:01.303684    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4addb3c-90c4-4813-9603-f28e8fa8be76-maas-api-tls\") pod \"maas-api-e2e-derive-19551b81-d9cf7f57d-qvmlr\" (UID: \"f4addb3c-90c4-4813-9603-f28e8fa8be76\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-19551b81-d9cf7f57d-qvmlr"
Jul 29 15:05:01.303906 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:01.303734    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4addb3c-90c4-4813-9603-f28e8fa8be76-openshift-service-ca\") pod \"maas-api-e2e-derive-19551b81-d9cf7f57d-qvmlr\" (UID: \"f4addb3c-90c4-4813-9603-f28e8fa8be76\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-19551b81-d9cf7f57d-qvmlr"
Jul 29 15:05:01.303906 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:01.303757    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x26gg\" (UniqueName: \"kubernetes.io/projected/f4addb3c-90c4-4813-9603-f28e8fa8be76-kube-api-access-x26gg\") pod \"maas-api-e2e-derive-19551b81-d9cf7f57d-qvmlr\" (UID: \"f4addb3c-90c4-4813-9603-f28e8fa8be76\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-19551b81-d9cf7f57d-qvmlr"
Jul 29 15:05:01.405265 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:01.405228    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4addb3c-90c4-4813-9603-f28e8fa8be76-maas-api-tls\") pod \"maas-api-e2e-derive-19551b81-d9cf7f57d-qvmlr\" (UID: \"f4addb3c-90c4-4813-9603-f28e8fa8be76\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-19551b81-d9cf7f57d-qvmlr"
Jul 29 15:05:01.405430 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:01.405277    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4addb3c-90c4-4813-9603-f28e8fa8be76-openshift-service-ca\") pod \"maas-api-e2e-derive-19551b81-d9cf7f57d-qvmlr\" (UID: \"f4addb3c-90c4-4813-9603-f28e8fa8be76\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-19551b81-d9cf7f57d-qvmlr"
Jul 29 15:05:01.405430 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:01.405307    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x26gg\" (UniqueName: \"kubernetes.io/projected/f4addb3c-90c4-4813-9603-f28e8fa8be76-kube-api-access-x26gg\") pod \"maas-api-e2e-derive-19551b81-d9cf7f57d-qvmlr\" (UID: \"f4addb3c-90c4-4813-9603-f28e8fa8be76\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-19551b81-d9cf7f57d-qvmlr"
Jul 29 15:05:01.405430 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:05:01.405383    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-19551b81-serving-cert: secret "maas-api-e2e-derive-19551b81-serving-cert" not found
Jul 29 15:05:01.405595 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:05:01.405447    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4addb3c-90c4-4813-9603-f28e8fa8be76-maas-api-tls podName:f4addb3c-90c4-4813-9603-f28e8fa8be76 nodeName:}" failed. No retries permitted until 2026-07-29 15:05:01.90542581 +0000 UTC m=+1803.953883743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f4addb3c-90c4-4813-9603-f28e8fa8be76-maas-api-tls") pod "maas-api-e2e-derive-19551b81-d9cf7f57d-qvmlr" (UID: "f4addb3c-90c4-4813-9603-f28e8fa8be76") : secret "maas-api-e2e-derive-19551b81-serving-cert" not found
Jul 29 15:05:01.406012 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:01.405976    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4addb3c-90c4-4813-9603-f28e8fa8be76-openshift-service-ca\") pod \"maas-api-e2e-derive-19551b81-d9cf7f57d-qvmlr\" (UID: \"f4addb3c-90c4-4813-9603-f28e8fa8be76\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-19551b81-d9cf7f57d-qvmlr"
Jul 29 15:05:01.416201 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:01.416180    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x26gg\" (UniqueName: \"kubernetes.io/projected/f4addb3c-90c4-4813-9603-f28e8fa8be76-kube-api-access-x26gg\") pod \"maas-api-e2e-derive-19551b81-d9cf7f57d-qvmlr\" (UID: \"f4addb3c-90c4-4813-9603-f28e8fa8be76\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-19551b81-d9cf7f57d-qvmlr"
Jul 29 15:05:01.911377 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:01.911342    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4addb3c-90c4-4813-9603-f28e8fa8be76-maas-api-tls\") pod \"maas-api-e2e-derive-19551b81-d9cf7f57d-qvmlr\" (UID: \"f4addb3c-90c4-4813-9603-f28e8fa8be76\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-19551b81-d9cf7f57d-qvmlr"
Jul 29 15:05:01.913904 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:01.913868    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4addb3c-90c4-4813-9603-f28e8fa8be76-maas-api-tls\") pod \"maas-api-e2e-derive-19551b81-d9cf7f57d-qvmlr\" (UID: \"f4addb3c-90c4-4813-9603-f28e8fa8be76\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-19551b81-d9cf7f57d-qvmlr"
Jul 29 15:05:02.112870 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:02.112833    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-19551b81-d9cf7f57d-qvmlr"
Jul 29 15:05:02.270328 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:02.270293    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-19551b81-d9cf7f57d-qvmlr"]
Jul 29 15:05:02.274758 ip-10-0-136-196 kubenswrapper[2563]: W0729 15:05:02.274730    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4addb3c_90c4_4813_9603_f28e8fa8be76.slice/crio-d55a7e8b4e2b294f4770ff7b22490df9a8769f7d72580bdf25f6a3a8b725641b WatchSource:0}: Error finding container d55a7e8b4e2b294f4770ff7b22490df9a8769f7d72580bdf25f6a3a8b725641b: Status 404 returned error can't find the container with id d55a7e8b4e2b294f4770ff7b22490df9a8769f7d72580bdf25f6a3a8b725641b
Jul 29 15:05:02.731051 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:02.731015    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-19551b81-d9cf7f57d-qvmlr" event={"ID":"f4addb3c-90c4-4813-9603-f28e8fa8be76","Type":"ContainerStarted","Data":"d55a7e8b4e2b294f4770ff7b22490df9a8769f7d72580bdf25f6a3a8b725641b"}
Jul 29 15:05:03.737912 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:03.737875    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-19551b81-d9cf7f57d-qvmlr" event={"ID":"f4addb3c-90c4-4813-9603-f28e8fa8be76","Type":"ContainerStarted","Data":"9fa16ddb39e14c0e05390264d35c0d66fe3308308c7532724e2435af86c1f553"}
Jul 29 15:05:03.738381 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:03.738019    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-19551b81-d9cf7f57d-qvmlr"
Jul 29 15:05:03.762391 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:03.762340    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-19551b81-d9cf7f57d-qvmlr" podStartSLOduration=2.085533891 podStartE2EDuration="2.76232703s" totalImagesPullingTime="676.793139ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:05:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:05:03.760605529 +0000 UTC m=+1805.809063469" watchObservedRunningTime="2026-07-29 15:05:03.76232703 +0000 UTC m=+1805.810784968"
Jul 29 15:05:09.754201 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:09.754166    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-19551b81-d9cf7f57d-qvmlr"
Jul 29 15:05:16.436434 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:16.435770    2563 patch_prober.go:28] interesting pod/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": read tcp 10.132.0.2:36594->10.132.0.52:15021: read: connection reset by peer" start-of-body=
Jul 29 15:05:16.436434 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:16.435833    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq" podUID="fd90a641-bffb-4137-aacb-4d8848864bb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": read tcp 10.132.0.2:36594->10.132.0.52:15021: read: connection reset by peer"
Jul 29 15:05:16.805078 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:16.805041    2563 generic.go:356] "Generic (PLEG): container finished" podID="fd90a641-bffb-4137-aacb-4d8848864bb8" containerID="26380c30eb9f7acbcd692d6ba7ef93f8399e9b798a5dbba9db714e466547c0c5" exitCode=0
Jul 29 15:05:16.805266 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:16.805104    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq" event={"ID":"fd90a641-bffb-4137-aacb-4d8848864bb8","Type":"ContainerDied","Data":"26380c30eb9f7acbcd692d6ba7ef93f8399e9b798a5dbba9db714e466547c0c5"}
Jul 29 15:05:16.805514 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:16.805499    2563 scope.go:117] "RemoveContainer" containerID="26380c30eb9f7acbcd692d6ba7ef93f8399e9b798a5dbba9db714e466547c0c5"
Jul 29 15:05:16.807464 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:16.807433    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:05:16.807544 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:16.807504    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:05:16.807544 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:16.807531    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:05:17.811795 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:17.811757    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq" event={"ID":"fd90a641-bffb-4137-aacb-4d8848864bb8","Type":"ContainerStarted","Data":"e14e0087248fe6e220bdba7b4f1f25b9f2a88432887913851616ca8a9f16cdf0"}
Jul 29 15:05:17.840355 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:17.840325    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"
Jul 29 15:05:17.840495 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:17.840443    2563 patch_prober.go:28] interesting pod/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Jul 29 15:05:17.840551 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:17.840489    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq" podUID="fd90a641-bffb-4137-aacb-4d8848864bb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Jul 29 15:05:18.840614 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:18.840580    2563 patch_prober.go:28] interesting pod/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Jul 29 15:05:18.841089 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:18.840643    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq" podUID="fd90a641-bffb-4137-aacb-4d8848864bb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Jul 29 15:05:19.840417 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:19.840379    2563 patch_prober.go:28] interesting pod/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Jul 29 15:05:19.840610 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:19.840437    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq" podUID="fd90a641-bffb-4137-aacb-4d8848864bb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Jul 29 15:05:20.840289 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:20.840253    2563 patch_prober.go:28] interesting pod/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Jul 29 15:05:20.840757 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:20.840314    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq" podUID="fd90a641-bffb-4137-aacb-4d8848864bb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Jul 29 15:05:21.844186 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:21.844156    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"
Jul 29 15:05:21.844631 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:21.844487    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"
Jul 29 15:05:21.845071 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:21.845057    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"
Jul 29 15:05:26.089635 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:26.089602    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-19551b81-dkd59"]
Jul 29 15:05:26.095316 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:26.095296    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-19551b81-dkd59"
Jul 29 15:05:26.097375 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:26.097350    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mmt6p\""
Jul 29 15:05:26.100940 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:26.100909    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-19551b81-dkd59"]
Jul 29 15:05:26.127554 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:26.127517    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5730b82e-5324-40ce-b11f-5348ddece967-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-19551b81-dkd59\" (UID: \"5730b82e-5324-40ce-b11f-5348ddece967\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-19551b81-dkd59"
Jul 29 15:05:26.228178 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:26.228111    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5730b82e-5324-40ce-b11f-5348ddece967-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-19551b81-dkd59\" (UID: \"5730b82e-5324-40ce-b11f-5348ddece967\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-19551b81-dkd59"
Jul 29 15:05:26.228844 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:26.228795    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5730b82e-5324-40ce-b11f-5348ddece967-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-19551b81-dkd59\" (UID: \"5730b82e-5324-40ce-b11f-5348ddece967\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-19551b81-dkd59"
Jul 29 15:05:26.409278 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:26.409195    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-19551b81-dkd59"
Jul 29 15:05:26.545185 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:26.545160    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-19551b81-dkd59"]
Jul 29 15:05:26.548574 ip-10-0-136-196 kubenswrapper[2563]: W0729 15:05:26.548538    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5730b82e_5324_40ce_b11f_5348ddece967.slice/crio-7d7bed87beb7058442785e39fa8d3673bbf8ae4d8424a8c116da21d3844eb850 WatchSource:0}: Error finding container 7d7bed87beb7058442785e39fa8d3673bbf8ae4d8424a8c116da21d3844eb850: Status 404 returned error can't find the container with id 7d7bed87beb7058442785e39fa8d3673bbf8ae4d8424a8c116da21d3844eb850
Jul 29 15:05:26.856264 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:26.856230    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-19551b81-dkd59" event={"ID":"5730b82e-5324-40ce-b11f-5348ddece967","Type":"ContainerStarted","Data":"7d7bed87beb7058442785e39fa8d3673bbf8ae4d8424a8c116da21d3844eb850"}
Jul 29 15:05:27.863894 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:27.863853    2563 generic.go:356] "Generic (PLEG): container finished" podID="5730b82e-5324-40ce-b11f-5348ddece967" containerID="3813bdfd8e0c28ca837cb68453b48977e149ca0ce0a67b0a96042eb67f6ef78c" exitCode=0
Jul 29 15:05:27.864392 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:27.863908    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-19551b81-dkd59" event={"ID":"5730b82e-5324-40ce-b11f-5348ddece967","Type":"ContainerDied","Data":"3813bdfd8e0c28ca837cb68453b48977e149ca0ce0a67b0a96042eb67f6ef78c"}
Jul 29 15:05:29.002342 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:29.002314    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-19551b81-dkd59"
Jul 29 15:05:29.052009 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:29.051977    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5730b82e-5324-40ce-b11f-5348ddece967-maas-api-service-ca\") pod \"5730b82e-5324-40ce-b11f-5348ddece967\" (UID: \"5730b82e-5324-40ce-b11f-5348ddece967\") "
Jul 29 15:05:29.052439 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:29.052411    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5730b82e-5324-40ce-b11f-5348ddece967-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5730b82e-5324-40ce-b11f-5348ddece967" (UID: "5730b82e-5324-40ce-b11f-5348ddece967"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 15:05:29.153520 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:29.153438    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5730b82e-5324-40ce-b11f-5348ddece967-maas-api-service-ca\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:05:29.873784 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:29.873748    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-19551b81-dkd59" event={"ID":"5730b82e-5324-40ce-b11f-5348ddece967","Type":"ContainerDied","Data":"7d7bed87beb7058442785e39fa8d3673bbf8ae4d8424a8c116da21d3844eb850"}
Jul 29 15:05:29.873784 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:29.873782    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-19551b81-dkd59"
Jul 29 15:05:29.874010 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:29.873801    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d7bed87beb7058442785e39fa8d3673bbf8ae4d8424a8c116da21d3844eb850"
Jul 29 15:05:31.778499 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:31.778471    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-19551b81-d9cf7f57d-qvmlr"]
Jul 29 15:05:31.778917 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:31.778666    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-19551b81-d9cf7f57d-qvmlr" podUID="f4addb3c-90c4-4813-9603-f28e8fa8be76" containerName="maas-api" containerID="cri-o://9fa16ddb39e14c0e05390264d35c0d66fe3308308c7532724e2435af86c1f553" gracePeriod=30
Jul 29 15:05:31.990621 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:31.990585    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-19551b81-dkd59"]
Jul 29 15:05:31.993882 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:31.993800    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-19551b81-dkd59"]
Jul 29 15:05:32.050590 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:32.050557    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-19551b81-d9cf7f57d-qvmlr"
Jul 29 15:05:32.079158 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:32.079104    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4addb3c-90c4-4813-9603-f28e8fa8be76-openshift-service-ca\") pod \"f4addb3c-90c4-4813-9603-f28e8fa8be76\" (UID: \"f4addb3c-90c4-4813-9603-f28e8fa8be76\") "
Jul 29 15:05:32.079331 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:32.079212    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4addb3c-90c4-4813-9603-f28e8fa8be76-maas-api-tls\") pod \"f4addb3c-90c4-4813-9603-f28e8fa8be76\" (UID: \"f4addb3c-90c4-4813-9603-f28e8fa8be76\") "
Jul 29 15:05:32.079331 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:32.079238    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x26gg\" (UniqueName: \"kubernetes.io/projected/f4addb3c-90c4-4813-9603-f28e8fa8be76-kube-api-access-x26gg\") pod \"f4addb3c-90c4-4813-9603-f28e8fa8be76\" (UID: \"f4addb3c-90c4-4813-9603-f28e8fa8be76\") "
Jul 29 15:05:32.079496 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:32.079467    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4addb3c-90c4-4813-9603-f28e8fa8be76-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f4addb3c-90c4-4813-9603-f28e8fa8be76" (UID: "f4addb3c-90c4-4813-9603-f28e8fa8be76"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 15:05:32.081317 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:32.081288    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4addb3c-90c4-4813-9603-f28e8fa8be76-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f4addb3c-90c4-4813-9603-f28e8fa8be76" (UID: "f4addb3c-90c4-4813-9603-f28e8fa8be76"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 15:05:32.081422 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:32.081320    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4addb3c-90c4-4813-9603-f28e8fa8be76-kube-api-access-x26gg" (OuterVolumeSpecName: "kube-api-access-x26gg") pod "f4addb3c-90c4-4813-9603-f28e8fa8be76" (UID: "f4addb3c-90c4-4813-9603-f28e8fa8be76"). InnerVolumeSpecName "kube-api-access-x26gg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 15:05:32.180098 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:32.180058    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4addb3c-90c4-4813-9603-f28e8fa8be76-openshift-service-ca\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:05:32.180098 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:32.180092    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4addb3c-90c4-4813-9603-f28e8fa8be76-maas-api-tls\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:05:32.180098 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:32.180102    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x26gg\" (UniqueName: \"kubernetes.io/projected/f4addb3c-90c4-4813-9603-f28e8fa8be76-kube-api-access-x26gg\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:05:32.351031 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:32.350933    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"]
Jul 29 15:05:32.351230 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:32.351198    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq" podUID="fd90a641-bffb-4137-aacb-4d8848864bb8" containerName="istio-proxy" containerID="cri-o://e14e0087248fe6e220bdba7b4f1f25b9f2a88432887913851616ca8a9f16cdf0" gracePeriod=30
Jul 29 15:05:32.432146 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:32.432103    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5730b82e-5324-40ce-b11f-5348ddece967" path="/var/lib/kubelet/pods/5730b82e-5324-40ce-b11f-5348ddece967/volumes"
Jul 29 15:05:32.888325 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:32.888289    2563 generic.go:356] "Generic (PLEG): container finished" podID="f4addb3c-90c4-4813-9603-f28e8fa8be76" containerID="9fa16ddb39e14c0e05390264d35c0d66fe3308308c7532724e2435af86c1f553" exitCode=0
Jul 29 15:05:32.888807 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:32.888347    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-19551b81-d9cf7f57d-qvmlr"
Jul 29 15:05:32.888807 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:32.888382    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-19551b81-d9cf7f57d-qvmlr" event={"ID":"f4addb3c-90c4-4813-9603-f28e8fa8be76","Type":"ContainerDied","Data":"9fa16ddb39e14c0e05390264d35c0d66fe3308308c7532724e2435af86c1f553"}
Jul 29 15:05:32.888807 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:32.888446    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-19551b81-d9cf7f57d-qvmlr" event={"ID":"f4addb3c-90c4-4813-9603-f28e8fa8be76","Type":"ContainerDied","Data":"d55a7e8b4e2b294f4770ff7b22490df9a8769f7d72580bdf25f6a3a8b725641b"}
Jul 29 15:05:32.888807 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:32.888485    2563 scope.go:117] "RemoveContainer" containerID="9fa16ddb39e14c0e05390264d35c0d66fe3308308c7532724e2435af86c1f553"
Jul 29 15:05:32.897263 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:32.897246    2563 scope.go:117] "RemoveContainer" containerID="9fa16ddb39e14c0e05390264d35c0d66fe3308308c7532724e2435af86c1f553"
Jul 29 15:05:32.897524 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:05:32.897499    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fa16ddb39e14c0e05390264d35c0d66fe3308308c7532724e2435af86c1f553\": container with ID starting with 9fa16ddb39e14c0e05390264d35c0d66fe3308308c7532724e2435af86c1f553 not found: ID does not exist" containerID="9fa16ddb39e14c0e05390264d35c0d66fe3308308c7532724e2435af86c1f553"
Jul 29 15:05:32.897575 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:32.897531    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fa16ddb39e14c0e05390264d35c0d66fe3308308c7532724e2435af86c1f553"} err="failed to get container status \"9fa16ddb39e14c0e05390264d35c0d66fe3308308c7532724e2435af86c1f553\": rpc error: code = NotFound desc = could not find container \"9fa16ddb39e14c0e05390264d35c0d66fe3308308c7532724e2435af86c1f553\": container with ID starting with 9fa16ddb39e14c0e05390264d35c0d66fe3308308c7532724e2435af86c1f553 not found: ID does not exist"
Jul 29 15:05:32.904191 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:32.904169    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-19551b81-d9cf7f57d-qvmlr"]
Jul 29 15:05:32.907737 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:32.907718    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-19551b81-d9cf7f57d-qvmlr"]
Jul 29 15:05:34.432615 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:34.432585    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4addb3c-90c4-4813-9603-f28e8fa8be76" path="/var/lib/kubelet/pods/f4addb3c-90c4-4813-9603-f28e8fa8be76/volumes"
Jul 29 15:05:36.846327 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:36.846294    2563 patch_prober.go:28] interesting pod/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 29 15:05:36.846690 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:36.846380    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq" podUID="fd90a641-bffb-4137-aacb-4d8848864bb8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 15:05:37.610287 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:37.610264    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"
Jul 29 15:05:37.725728 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:37.725705    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd90a641-bffb-4137-aacb-4d8848864bb8-istio-token\") pod \"fd90a641-bffb-4137-aacb-4d8848864bb8\" (UID: \"fd90a641-bffb-4137-aacb-4d8848864bb8\") "
Jul 29 15:05:37.725908 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:37.725739    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd90a641-bffb-4137-aacb-4d8848864bb8-istio-data\") pod \"fd90a641-bffb-4137-aacb-4d8848864bb8\" (UID: \"fd90a641-bffb-4137-aacb-4d8848864bb8\") "
Jul 29 15:05:37.725908 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:37.725772    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd90a641-bffb-4137-aacb-4d8848864bb8-credential-socket\") pod \"fd90a641-bffb-4137-aacb-4d8848864bb8\" (UID: \"fd90a641-bffb-4137-aacb-4d8848864bb8\") "
Jul 29 15:05:37.725908 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:37.725790    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd90a641-bffb-4137-aacb-4d8848864bb8-workload-certs\") pod \"fd90a641-bffb-4137-aacb-4d8848864bb8\" (UID: \"fd90a641-bffb-4137-aacb-4d8848864bb8\") "
Jul 29 15:05:37.726086 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:37.725989    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9lhx\" (UniqueName: \"kubernetes.io/projected/fd90a641-bffb-4137-aacb-4d8848864bb8-kube-api-access-r9lhx\") pod \"fd90a641-bffb-4137-aacb-4d8848864bb8\" (UID: \"fd90a641-bffb-4137-aacb-4d8848864bb8\") "
Jul 29 15:05:37.726086 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:37.726019    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd90a641-bffb-4137-aacb-4d8848864bb8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fd90a641-bffb-4137-aacb-4d8848864bb8" (UID: "fd90a641-bffb-4137-aacb-4d8848864bb8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:05:37.726086 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:37.726031    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd90a641-bffb-4137-aacb-4d8848864bb8-istiod-ca-cert\") pod \"fd90a641-bffb-4137-aacb-4d8848864bb8\" (UID: \"fd90a641-bffb-4137-aacb-4d8848864bb8\") "
Jul 29 15:05:37.726267 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:37.726085    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd90a641-bffb-4137-aacb-4d8848864bb8-workload-socket\") pod \"fd90a641-bffb-4137-aacb-4d8848864bb8\" (UID: \"fd90a641-bffb-4137-aacb-4d8848864bb8\") "
Jul 29 15:05:37.726267 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:37.726085    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd90a641-bffb-4137-aacb-4d8848864bb8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fd90a641-bffb-4137-aacb-4d8848864bb8" (UID: "fd90a641-bffb-4137-aacb-4d8848864bb8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:05:37.726267 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:37.726207    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd90a641-bffb-4137-aacb-4d8848864bb8-istio-podinfo\") pod \"fd90a641-bffb-4137-aacb-4d8848864bb8\" (UID: \"fd90a641-bffb-4137-aacb-4d8848864bb8\") "
Jul 29 15:05:37.726443 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:37.726280    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd90a641-bffb-4137-aacb-4d8848864bb8-istio-envoy\") pod \"fd90a641-bffb-4137-aacb-4d8848864bb8\" (UID: \"fd90a641-bffb-4137-aacb-4d8848864bb8\") "
Jul 29 15:05:37.726443 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:37.726281    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd90a641-bffb-4137-aacb-4d8848864bb8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fd90a641-bffb-4137-aacb-4d8848864bb8" (UID: "fd90a641-bffb-4137-aacb-4d8848864bb8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:05:37.726443 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:37.726385    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd90a641-bffb-4137-aacb-4d8848864bb8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fd90a641-bffb-4137-aacb-4d8848864bb8" (UID: "fd90a641-bffb-4137-aacb-4d8848864bb8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 15:05:37.726619 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:37.726535    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd90a641-bffb-4137-aacb-4d8848864bb8-credential-socket\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:05:37.726619 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:37.726552    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd90a641-bffb-4137-aacb-4d8848864bb8-workload-certs\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:05:37.726619 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:37.726566    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd90a641-bffb-4137-aacb-4d8848864bb8-istiod-ca-cert\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:05:37.726619 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:37.726580    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd90a641-bffb-4137-aacb-4d8848864bb8-workload-socket\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:05:37.726875 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:37.726850    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd90a641-bffb-4137-aacb-4d8848864bb8-istio-data" (OuterVolumeSpecName: "istio-data") pod "fd90a641-bffb-4137-aacb-4d8848864bb8" (UID: "fd90a641-bffb-4137-aacb-4d8848864bb8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:05:37.728228 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:37.728207    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd90a641-bffb-4137-aacb-4d8848864bb8-kube-api-access-r9lhx" (OuterVolumeSpecName: "kube-api-access-r9lhx") pod "fd90a641-bffb-4137-aacb-4d8848864bb8" (UID: "fd90a641-bffb-4137-aacb-4d8848864bb8"). InnerVolumeSpecName "kube-api-access-r9lhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 15:05:37.728326 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:37.728311    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd90a641-bffb-4137-aacb-4d8848864bb8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fd90a641-bffb-4137-aacb-4d8848864bb8" (UID: "fd90a641-bffb-4137-aacb-4d8848864bb8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:05:37.728443 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:37.728427    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd90a641-bffb-4137-aacb-4d8848864bb8-istio-token" (OuterVolumeSpecName: "istio-token") pod "fd90a641-bffb-4137-aacb-4d8848864bb8" (UID: "fd90a641-bffb-4137-aacb-4d8848864bb8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 15:05:37.728520 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:37.728506    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd90a641-bffb-4137-aacb-4d8848864bb8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fd90a641-bffb-4137-aacb-4d8848864bb8" (UID: "fd90a641-bffb-4137-aacb-4d8848864bb8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 15:05:37.827054 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:37.827023    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd90a641-bffb-4137-aacb-4d8848864bb8-istio-token\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:05:37.827054 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:37.827049    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd90a641-bffb-4137-aacb-4d8848864bb8-istio-data\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:05:37.827260 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:37.827062    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9lhx\" (UniqueName: \"kubernetes.io/projected/fd90a641-bffb-4137-aacb-4d8848864bb8-kube-api-access-r9lhx\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:05:37.827260 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:37.827075    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd90a641-bffb-4137-aacb-4d8848864bb8-istio-podinfo\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:05:37.827260 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:37.827089    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd90a641-bffb-4137-aacb-4d8848864bb8-istio-envoy\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:05:37.915430 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:37.915395    2563 generic.go:356] "Generic (PLEG): container finished" podID="fd90a641-bffb-4137-aacb-4d8848864bb8" containerID="e14e0087248fe6e220bdba7b4f1f25b9f2a88432887913851616ca8a9f16cdf0" exitCode=0
Jul 29 15:05:37.915791 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:37.915454    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"
Jul 29 15:05:37.915791 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:37.915490    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq" event={"ID":"fd90a641-bffb-4137-aacb-4d8848864bb8","Type":"ContainerDied","Data":"e14e0087248fe6e220bdba7b4f1f25b9f2a88432887913851616ca8a9f16cdf0"}
Jul 29 15:05:37.915791 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:37.915573    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq" event={"ID":"fd90a641-bffb-4137-aacb-4d8848864bb8","Type":"ContainerDied","Data":"49c77db8684d703d05cc5d616a51f856b5bd4e84b98b52464b44e6536faa488b"}
Jul 29 15:05:37.915791 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:37.915596    2563 scope.go:117] "RemoveContainer" containerID="e14e0087248fe6e220bdba7b4f1f25b9f2a88432887913851616ca8a9f16cdf0"
Jul 29 15:05:37.925057 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:37.925037    2563 scope.go:117] "RemoveContainer" containerID="26380c30eb9f7acbcd692d6ba7ef93f8399e9b798a5dbba9db714e466547c0c5"
Jul 29 15:05:37.932432 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:37.932413    2563 scope.go:117] "RemoveContainer" containerID="e14e0087248fe6e220bdba7b4f1f25b9f2a88432887913851616ca8a9f16cdf0"
Jul 29 15:05:37.932660 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:05:37.932641    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e14e0087248fe6e220bdba7b4f1f25b9f2a88432887913851616ca8a9f16cdf0\": container with ID starting with e14e0087248fe6e220bdba7b4f1f25b9f2a88432887913851616ca8a9f16cdf0 not found: ID does not exist" containerID="e14e0087248fe6e220bdba7b4f1f25b9f2a88432887913851616ca8a9f16cdf0"
Jul 29 15:05:37.932718 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:37.932668    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e14e0087248fe6e220bdba7b4f1f25b9f2a88432887913851616ca8a9f16cdf0"} err="failed to get container status \"e14e0087248fe6e220bdba7b4f1f25b9f2a88432887913851616ca8a9f16cdf0\": rpc error: code = NotFound desc = could not find container \"e14e0087248fe6e220bdba7b4f1f25b9f2a88432887913851616ca8a9f16cdf0\": container with ID starting with e14e0087248fe6e220bdba7b4f1f25b9f2a88432887913851616ca8a9f16cdf0 not found: ID does not exist"
Jul 29 15:05:37.932718 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:37.932684    2563 scope.go:117] "RemoveContainer" containerID="26380c30eb9f7acbcd692d6ba7ef93f8399e9b798a5dbba9db714e466547c0c5"
Jul 29 15:05:37.932908 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:05:37.932893    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26380c30eb9f7acbcd692d6ba7ef93f8399e9b798a5dbba9db714e466547c0c5\": container with ID starting with 26380c30eb9f7acbcd692d6ba7ef93f8399e9b798a5dbba9db714e466547c0c5 not found: ID does not exist" containerID="26380c30eb9f7acbcd692d6ba7ef93f8399e9b798a5dbba9db714e466547c0c5"
Jul 29 15:05:37.932955 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:37.932910    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26380c30eb9f7acbcd692d6ba7ef93f8399e9b798a5dbba9db714e466547c0c5"} err="failed to get container status \"26380c30eb9f7acbcd692d6ba7ef93f8399e9b798a5dbba9db714e466547c0c5\": rpc error: code = NotFound desc = could not find container \"26380c30eb9f7acbcd692d6ba7ef93f8399e9b798a5dbba9db714e466547c0c5\": container with ID starting with 26380c30eb9f7acbcd692d6ba7ef93f8399e9b798a5dbba9db714e466547c0c5 not found: ID does not exist"
Jul 29 15:05:37.938458 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:37.938436    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"]
Jul 29 15:05:37.941504 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:37.941485    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-19551b81-openshift-default-84879cf4c6-c9rvq"]
Jul 29 15:05:38.432374 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:05:38.432337    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd90a641-bffb-4137-aacb-4d8848864bb8" path="/var/lib/kubelet/pods/fd90a641-bffb-4137-aacb-4d8848864bb8/volumes"
Jul 29 15:07:21.742634 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.742599    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"]
Jul 29 15:07:21.743100 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.742955    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5730b82e-5324-40ce-b11f-5348ddece967" containerName="revoke-keys"
Jul 29 15:07:21.743100 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.742965    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5730b82e-5324-40ce-b11f-5348ddece967" containerName="revoke-keys"
Jul 29 15:07:21.743100 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.742985    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd90a641-bffb-4137-aacb-4d8848864bb8" containerName="istio-proxy"
Jul 29 15:07:21.743100 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.742991    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd90a641-bffb-4137-aacb-4d8848864bb8" containerName="istio-proxy"
Jul 29 15:07:21.743100 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.742999    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd90a641-bffb-4137-aacb-4d8848864bb8" containerName="istio-proxy"
Jul 29 15:07:21.743100 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.743004    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd90a641-bffb-4137-aacb-4d8848864bb8" containerName="istio-proxy"
Jul 29 15:07:21.743100 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.743011    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4addb3c-90c4-4813-9603-f28e8fa8be76" containerName="maas-api"
Jul 29 15:07:21.743100 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.743017    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4addb3c-90c4-4813-9603-f28e8fa8be76" containerName="maas-api"
Jul 29 15:07:21.743100 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.743069    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd90a641-bffb-4137-aacb-4d8848864bb8" containerName="istio-proxy"
Jul 29 15:07:21.743100 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.743078    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4addb3c-90c4-4813-9603-f28e8fa8be76" containerName="maas-api"
Jul 29 15:07:21.743100 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.743087    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5730b82e-5324-40ce-b11f-5348ddece967" containerName="revoke-keys"
Jul 29 15:07:21.746383 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.746353    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"
Jul 29 15:07:21.748481 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.748454    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 29 15:07:21.748597 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.748568    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-0af43e8c-openshift-default-dockercfg-pdv56\""
Jul 29 15:07:21.764042 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.764004    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"]
Jul 29 15:07:21.836058 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.836016    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-istiod-ca-cert\") pod \"e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x\" (UID: \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\") " pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"
Jul 29 15:07:21.836252 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.836064    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-workload-socket\") pod \"e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x\" (UID: \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\") " pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"
Jul 29 15:07:21.836252 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.836099    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-istio-podinfo\") pod \"e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x\" (UID: \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\") " pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"
Jul 29 15:07:21.836252 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.836160    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-istio-envoy\") pod \"e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x\" (UID: \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\") " pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"
Jul 29 15:07:21.836252 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.836194    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-workload-certs\") pod \"e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x\" (UID: \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\") " pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"
Jul 29 15:07:21.836416 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.836268    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-istio-token\") pod \"e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x\" (UID: \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\") " pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"
Jul 29 15:07:21.836416 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.836305    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-istio-data\") pod \"e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x\" (UID: \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\") " pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"
Jul 29 15:07:21.836416 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.836366    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf7wf\" (UniqueName: \"kubernetes.io/projected/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-kube-api-access-jf7wf\") pod \"e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x\" (UID: \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\") " pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"
Jul 29 15:07:21.836416 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.836406    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-credential-socket\") pod \"e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x\" (UID: \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\") " pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"
Jul 29 15:07:21.937649 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.937609    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf7wf\" (UniqueName: \"kubernetes.io/projected/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-kube-api-access-jf7wf\") pod \"e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x\" (UID: \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\") " pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"
Jul 29 15:07:21.937831 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.937663    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-credential-socket\") pod \"e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x\" (UID: \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\") " pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"
Jul 29 15:07:21.937831 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.937704    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-istiod-ca-cert\") pod \"e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x\" (UID: \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\") " pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"
Jul 29 15:07:21.937831 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.937739    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-workload-socket\") pod \"e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x\" (UID: \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\") " pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"
Jul 29 15:07:21.937831 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.937768    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-istio-podinfo\") pod \"e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x\" (UID: \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\") " pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"
Jul 29 15:07:21.937831 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.937799    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-istio-envoy\") pod \"e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x\" (UID: \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\") " pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"
Jul 29 15:07:21.937831 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.937828    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-workload-certs\") pod \"e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x\" (UID: \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\") " pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"
Jul 29 15:07:21.938214 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.937871    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-istio-token\") pod \"e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x\" (UID: \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\") " pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"
Jul 29 15:07:21.938214 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.937906    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-istio-data\") pod \"e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x\" (UID: \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\") " pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"
Jul 29 15:07:21.938214 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.938170    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-credential-socket\") pod \"e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x\" (UID: \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\") " pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"
Jul 29 15:07:21.938510 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.938338    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-workload-certs\") pod \"e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x\" (UID: \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\") " pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"
Jul 29 15:07:21.938510 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.938386    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-istio-data\") pod \"e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x\" (UID: \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\") " pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"
Jul 29 15:07:21.938629 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.938586    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-workload-socket\") pod \"e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x\" (UID: \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\") " pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"
Jul 29 15:07:21.938629 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.938607    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-istiod-ca-cert\") pod \"e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x\" (UID: \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\") " pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"
Jul 29 15:07:21.940297 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.940272    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-istio-envoy\") pod \"e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x\" (UID: \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\") " pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"
Jul 29 15:07:21.940412 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.940395    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-istio-podinfo\") pod \"e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x\" (UID: \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\") " pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"
Jul 29 15:07:21.953686 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.953654    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-istio-token\") pod \"e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x\" (UID: \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\") " pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"
Jul 29 15:07:21.955601 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:21.955581    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf7wf\" (UniqueName: \"kubernetes.io/projected/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-kube-api-access-jf7wf\") pod \"e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x\" (UID: \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\") " pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"
Jul 29 15:07:22.057356 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:22.057269    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"
Jul 29 15:07:22.191730 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:22.191702    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"]
Jul 29 15:07:22.193611 ip-10-0-136-196 kubenswrapper[2563]: W0729 15:07:22.193571    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9db1f2b9_0437_4cf3_a7d7_fb45b60f181c.slice/crio-75a25ccf2ae8cf1d27439effa9cdae455f52b5d1f97d26f7f71f473647edd253 WatchSource:0}: Error finding container 75a25ccf2ae8cf1d27439effa9cdae455f52b5d1f97d26f7f71f473647edd253: Status 404 returned error can't find the container with id 75a25ccf2ae8cf1d27439effa9cdae455f52b5d1f97d26f7f71f473647edd253
Jul 29 15:07:22.195652 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:22.195623    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:07:22.195720 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:22.195698    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:07:22.195763 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:22.195726    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:07:22.432497 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:22.432402    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x" event={"ID":"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c","Type":"ContainerStarted","Data":"7976f9c0d563ab8ff5811681267471a3e18e7bea84ce1572acd675f9f59c533c"}
Jul 29 15:07:22.432497 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:22.432462    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x" event={"ID":"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c","Type":"ContainerStarted","Data":"75a25ccf2ae8cf1d27439effa9cdae455f52b5d1f97d26f7f71f473647edd253"}
Jul 29 15:07:22.471890 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:22.471830    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x" podStartSLOduration=1.471806868 podStartE2EDuration="1.471806868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:07:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:07:22.468001831 +0000 UTC m=+1944.516459773" watchObservedRunningTime="2026-07-29 15:07:22.471806868 +0000 UTC m=+1944.520264808"
Jul 29 15:07:23.057405 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:23.057363    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"
Jul 29 15:07:23.062350 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:23.062326    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"
Jul 29 15:07:23.432792 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:23.432717    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"
Jul 29 15:07:23.433695 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:23.433674    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"
Jul 29 15:07:24.703863 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:24.703827    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0af43e8c-5d4f7bfc96-r824g"]
Jul 29 15:07:24.704706 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:24.704680    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd90a641-bffb-4137-aacb-4d8848864bb8" containerName="istio-proxy"
Jul 29 15:07:24.708230 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:24.708212    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0af43e8c-5d4f7bfc96-r824g"
Jul 29 15:07:24.711101 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:24.711081    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-0af43e8c-serving-cert\""
Jul 29 15:07:24.720617 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:24.720596    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0af43e8c-5d4f7bfc96-r824g"]
Jul 29 15:07:24.764563 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:24.764529    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2rmq\" (UniqueName: \"kubernetes.io/projected/dc5f8d60-4e46-42c2-9de0-e61ce88a1c84-kube-api-access-w2rmq\") pod \"maas-api-e2e-mt-0af43e8c-5d4f7bfc96-r824g\" (UID: \"dc5f8d60-4e46-42c2-9de0-e61ce88a1c84\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0af43e8c-5d4f7bfc96-r824g"
Jul 29 15:07:24.764758 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:24.764588    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc5f8d60-4e46-42c2-9de0-e61ce88a1c84-openshift-service-ca\") pod \"maas-api-e2e-mt-0af43e8c-5d4f7bfc96-r824g\" (UID: \"dc5f8d60-4e46-42c2-9de0-e61ce88a1c84\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0af43e8c-5d4f7bfc96-r824g"
Jul 29 15:07:24.764758 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:24.764703    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc5f8d60-4e46-42c2-9de0-e61ce88a1c84-maas-api-tls\") pod \"maas-api-e2e-mt-0af43e8c-5d4f7bfc96-r824g\" (UID: \"dc5f8d60-4e46-42c2-9de0-e61ce88a1c84\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0af43e8c-5d4f7bfc96-r824g"
Jul 29 15:07:24.866077 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:24.866036    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2rmq\" (UniqueName: \"kubernetes.io/projected/dc5f8d60-4e46-42c2-9de0-e61ce88a1c84-kube-api-access-w2rmq\") pod \"maas-api-e2e-mt-0af43e8c-5d4f7bfc96-r824g\" (UID: \"dc5f8d60-4e46-42c2-9de0-e61ce88a1c84\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0af43e8c-5d4f7bfc96-r824g"
Jul 29 15:07:24.866268 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:24.866116    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc5f8d60-4e46-42c2-9de0-e61ce88a1c84-openshift-service-ca\") pod \"maas-api-e2e-mt-0af43e8c-5d4f7bfc96-r824g\" (UID: \"dc5f8d60-4e46-42c2-9de0-e61ce88a1c84\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0af43e8c-5d4f7bfc96-r824g"
Jul 29 15:07:24.866268 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:24.866209    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc5f8d60-4e46-42c2-9de0-e61ce88a1c84-maas-api-tls\") pod \"maas-api-e2e-mt-0af43e8c-5d4f7bfc96-r824g\" (UID: \"dc5f8d60-4e46-42c2-9de0-e61ce88a1c84\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0af43e8c-5d4f7bfc96-r824g"
Jul 29 15:07:24.866411 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:07:24.866377    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-0af43e8c-serving-cert: secret "maas-api-e2e-mt-0af43e8c-serving-cert" not found
Jul 29 15:07:24.866502 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:07:24.866490    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc5f8d60-4e46-42c2-9de0-e61ce88a1c84-maas-api-tls podName:dc5f8d60-4e46-42c2-9de0-e61ce88a1c84 nodeName:}" failed. No retries permitted until 2026-07-29 15:07:25.366468432 +0000 UTC m=+1947.414926364 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dc5f8d60-4e46-42c2-9de0-e61ce88a1c84-maas-api-tls") pod "maas-api-e2e-mt-0af43e8c-5d4f7bfc96-r824g" (UID: "dc5f8d60-4e46-42c2-9de0-e61ce88a1c84") : secret "maas-api-e2e-mt-0af43e8c-serving-cert" not found
Jul 29 15:07:24.866813 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:24.866794    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc5f8d60-4e46-42c2-9de0-e61ce88a1c84-openshift-service-ca\") pod \"maas-api-e2e-mt-0af43e8c-5d4f7bfc96-r824g\" (UID: \"dc5f8d60-4e46-42c2-9de0-e61ce88a1c84\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0af43e8c-5d4f7bfc96-r824g"
Jul 29 15:07:24.883094 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:24.883070    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2rmq\" (UniqueName: \"kubernetes.io/projected/dc5f8d60-4e46-42c2-9de0-e61ce88a1c84-kube-api-access-w2rmq\") pod \"maas-api-e2e-mt-0af43e8c-5d4f7bfc96-r824g\" (UID: \"dc5f8d60-4e46-42c2-9de0-e61ce88a1c84\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0af43e8c-5d4f7bfc96-r824g"
Jul 29 15:07:25.371284 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:25.371251    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc5f8d60-4e46-42c2-9de0-e61ce88a1c84-maas-api-tls\") pod \"maas-api-e2e-mt-0af43e8c-5d4f7bfc96-r824g\" (UID: \"dc5f8d60-4e46-42c2-9de0-e61ce88a1c84\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0af43e8c-5d4f7bfc96-r824g"
Jul 29 15:07:25.373598 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:25.373579    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc5f8d60-4e46-42c2-9de0-e61ce88a1c84-maas-api-tls\") pod \"maas-api-e2e-mt-0af43e8c-5d4f7bfc96-r824g\" (UID: \"dc5f8d60-4e46-42c2-9de0-e61ce88a1c84\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0af43e8c-5d4f7bfc96-r824g"
Jul 29 15:07:25.620473 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:25.620443    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0af43e8c-5d4f7bfc96-r824g"
Jul 29 15:07:25.758971 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:25.758931    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0af43e8c-5d4f7bfc96-r824g"]
Jul 29 15:07:25.763035 ip-10-0-136-196 kubenswrapper[2563]: W0729 15:07:25.762995    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc5f8d60_4e46_42c2_9de0_e61ce88a1c84.slice/crio-5254de87591230c5607acdf5412d80e300aa07ad581a783c293cac00ec9a7ab0 WatchSource:0}: Error finding container 5254de87591230c5607acdf5412d80e300aa07ad581a783c293cac00ec9a7ab0: Status 404 returned error can't find the container with id 5254de87591230c5607acdf5412d80e300aa07ad581a783c293cac00ec9a7ab0
Jul 29 15:07:26.447256 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:26.447220    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0af43e8c-5d4f7bfc96-r824g" event={"ID":"dc5f8d60-4e46-42c2-9de0-e61ce88a1c84","Type":"ContainerStarted","Data":"5254de87591230c5607acdf5412d80e300aa07ad581a783c293cac00ec9a7ab0"}
Jul 29 15:07:28.458398 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:28.458350    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0af43e8c-5d4f7bfc96-r824g" event={"ID":"dc5f8d60-4e46-42c2-9de0-e61ce88a1c84","Type":"ContainerStarted","Data":"2ce647db3b61172d2ae4abad4d49cd609e44ceaeac6270b565ad0cc17efe94cc"}
Jul 29 15:07:28.458765 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:28.458429    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0af43e8c-5d4f7bfc96-r824g"
Jul 29 15:07:28.476795 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:28.476734    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0af43e8c-5d4f7bfc96-r824g" podStartSLOduration=2.435543385 podStartE2EDuration="4.476718029s" totalImagesPullingTime="2.041174644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:07:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:07:28.474752004 +0000 UTC m=+1950.523209944" watchObservedRunningTime="2026-07-29 15:07:28.476718029 +0000 UTC m=+1950.525175999"
Jul 29 15:07:34.473693 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:34.473664    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0af43e8c-5d4f7bfc96-r824g"
Jul 29 15:07:38.434234 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:38.434203    2563 patch_prober.go:28] interesting pod/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Jul 29 15:07:38.434738 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:38.434271    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x" podUID="9db1f2b9-0437-4cf3-a7d7-fb45b60f181c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Jul 29 15:07:38.511119 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:38.511087    2563 generic.go:356] "Generic (PLEG): container finished" podID="9db1f2b9-0437-4cf3-a7d7-fb45b60f181c" containerID="7976f9c0d563ab8ff5811681267471a3e18e7bea84ce1572acd675f9f59c533c" exitCode=0
Jul 29 15:07:38.511297 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:38.511171    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x" event={"ID":"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c","Type":"ContainerDied","Data":"7976f9c0d563ab8ff5811681267471a3e18e7bea84ce1572acd675f9f59c533c"}
Jul 29 15:07:38.511549 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:38.511535    2563 scope.go:117] "RemoveContainer" containerID="7976f9c0d563ab8ff5811681267471a3e18e7bea84ce1572acd675f9f59c533c"
Jul 29 15:07:38.512518 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:38.512492    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:07:38.512617 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:38.512555    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:07:38.512617 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:38.512586    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:07:39.517519 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:39.517487    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x" event={"ID":"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c","Type":"ContainerStarted","Data":"e4abf47567da98cd46a417b2afae65050ad9dfe58fdd8489c9c444c9deca0a9c"}
Jul 29 15:07:40.057731 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:40.057678    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"
Jul 29 15:07:40.058201 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:40.058169    2563 patch_prober.go:28] interesting pod/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Jul 29 15:07:40.058432 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:40.058400    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x" podUID="9db1f2b9-0437-4cf3-a7d7-fb45b60f181c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Jul 29 15:07:41.058448 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:41.058412    2563 patch_prober.go:28] interesting pod/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Jul 29 15:07:41.059012 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:41.058981    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x" podUID="9db1f2b9-0437-4cf3-a7d7-fb45b60f181c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Jul 29 15:07:42.058235 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:42.058191    2563 patch_prober.go:28] interesting pod/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Jul 29 15:07:42.058454 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:42.058262    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x" podUID="9db1f2b9-0437-4cf3-a7d7-fb45b60f181c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Jul 29 15:07:43.059189 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:43.058689    2563 patch_prober.go:28] interesting pod/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Jul 29 15:07:43.059189 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:43.058769    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x" podUID="9db1f2b9-0437-4cf3-a7d7-fb45b60f181c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Jul 29 15:07:44.061626 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:44.061590    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"
Jul 29 15:07:44.061991 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:44.061880    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"
Jul 29 15:07:44.062611 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:44.062590    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"
Jul 29 15:07:57.006414 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:57.006319    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0af43e8c-5q2xz"]
Jul 29 15:07:57.010520 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:57.010496    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0af43e8c-5q2xz"
Jul 29 15:07:57.013511 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:57.013484    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mmt6p\""
Jul 29 15:07:57.019994 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:57.019970    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0af43e8c-5q2xz"]
Jul 29 15:07:57.073069 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:57.073035    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d27743fd-a4fd-4ffc-923c-8630819dfe5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0af43e8c-5q2xz\" (UID: \"d27743fd-a4fd-4ffc-923c-8630819dfe5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0af43e8c-5q2xz"
Jul 29 15:07:57.174216 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:57.174174    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d27743fd-a4fd-4ffc-923c-8630819dfe5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0af43e8c-5q2xz\" (UID: \"d27743fd-a4fd-4ffc-923c-8630819dfe5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0af43e8c-5q2xz"
Jul 29 15:07:57.174873 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:57.174845    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d27743fd-a4fd-4ffc-923c-8630819dfe5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0af43e8c-5q2xz\" (UID: \"d27743fd-a4fd-4ffc-923c-8630819dfe5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0af43e8c-5q2xz"
Jul 29 15:07:57.322808 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:57.322740    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0af43e8c-5q2xz"
Jul 29 15:07:57.459113 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:57.459085    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0af43e8c-5q2xz"]
Jul 29 15:07:57.461653 ip-10-0-136-196 kubenswrapper[2563]: W0729 15:07:57.461621    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd27743fd_a4fd_4ffc_923c_8630819dfe5a.slice/crio-e7c81ed99c77bf3f1857e6e85d7fac3c1d14fbf09f6dbde677568e8864fcfc5e WatchSource:0}: Error finding container e7c81ed99c77bf3f1857e6e85d7fac3c1d14fbf09f6dbde677568e8864fcfc5e: Status 404 returned error can't find the container with id e7c81ed99c77bf3f1857e6e85d7fac3c1d14fbf09f6dbde677568e8864fcfc5e
Jul 29 15:07:57.606626 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:57.606544    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0af43e8c-5q2xz" event={"ID":"d27743fd-a4fd-4ffc-923c-8630819dfe5a","Type":"ContainerStarted","Data":"e7c81ed99c77bf3f1857e6e85d7fac3c1d14fbf09f6dbde677568e8864fcfc5e"}
Jul 29 15:07:58.612672 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:58.612632    2563 generic.go:356] "Generic (PLEG): container finished" podID="d27743fd-a4fd-4ffc-923c-8630819dfe5a" containerID="e556cd9e28d5bc669599cb78ef678cdf223ddd6fca09c244d8a3feea9e4cefb1" exitCode=0
Jul 29 15:07:58.613047 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:58.612678    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0af43e8c-5q2xz" event={"ID":"d27743fd-a4fd-4ffc-923c-8630819dfe5a","Type":"ContainerDied","Data":"e556cd9e28d5bc669599cb78ef678cdf223ddd6fca09c244d8a3feea9e4cefb1"}
Jul 29 15:07:59.750492 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:59.750465    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0af43e8c-5q2xz"
Jul 29 15:07:59.796454 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:59.796418    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d27743fd-a4fd-4ffc-923c-8630819dfe5a-maas-api-service-ca\") pod \"d27743fd-a4fd-4ffc-923c-8630819dfe5a\" (UID: \"d27743fd-a4fd-4ffc-923c-8630819dfe5a\") "
Jul 29 15:07:59.796770 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:59.796748    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d27743fd-a4fd-4ffc-923c-8630819dfe5a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d27743fd-a4fd-4ffc-923c-8630819dfe5a" (UID: "d27743fd-a4fd-4ffc-923c-8630819dfe5a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 15:07:59.897756 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:07:59.897668    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d27743fd-a4fd-4ffc-923c-8630819dfe5a-maas-api-service-ca\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:08:00.624799 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:00.624763    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0af43e8c-5q2xz" event={"ID":"d27743fd-a4fd-4ffc-923c-8630819dfe5a","Type":"ContainerDied","Data":"e7c81ed99c77bf3f1857e6e85d7fac3c1d14fbf09f6dbde677568e8864fcfc5e"}
Jul 29 15:08:00.624799 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:00.624788    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0af43e8c-5q2xz"
Jul 29 15:08:00.624995 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:00.624818    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7c81ed99c77bf3f1857e6e85d7fac3c1d14fbf09f6dbde677568e8864fcfc5e"
Jul 29 15:08:07.486183 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:07.486147    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0af43e8c-5d4f7bfc96-r824g"]
Jul 29 15:08:07.486611 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:07.486381    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0af43e8c-5d4f7bfc96-r824g" podUID="dc5f8d60-4e46-42c2-9de0-e61ce88a1c84" containerName="maas-api" containerID="cri-o://2ce647db3b61172d2ae4abad4d49cd609e44ceaeac6270b565ad0cc17efe94cc" gracePeriod=30
Jul 29 15:08:07.668285 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:07.668247    2563 generic.go:356] "Generic (PLEG): container finished" podID="dc5f8d60-4e46-42c2-9de0-e61ce88a1c84" containerID="2ce647db3b61172d2ae4abad4d49cd609e44ceaeac6270b565ad0cc17efe94cc" exitCode=0
Jul 29 15:08:07.668429 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:07.668315    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0af43e8c-5d4f7bfc96-r824g" event={"ID":"dc5f8d60-4e46-42c2-9de0-e61ce88a1c84","Type":"ContainerDied","Data":"2ce647db3b61172d2ae4abad4d49cd609e44ceaeac6270b565ad0cc17efe94cc"}
Jul 29 15:08:07.709791 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:07.709759    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0af43e8c-5q2xz"]
Jul 29 15:08:07.715957 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:07.715931    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0af43e8c-5q2xz"]
Jul 29 15:08:07.740442 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:07.740381    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0af43e8c-5d4f7bfc96-r824g"
Jul 29 15:08:07.865666 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:07.865635    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc5f8d60-4e46-42c2-9de0-e61ce88a1c84-openshift-service-ca\") pod \"dc5f8d60-4e46-42c2-9de0-e61ce88a1c84\" (UID: \"dc5f8d60-4e46-42c2-9de0-e61ce88a1c84\") "
Jul 29 15:08:07.865873 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:07.865697    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc5f8d60-4e46-42c2-9de0-e61ce88a1c84-maas-api-tls\") pod \"dc5f8d60-4e46-42c2-9de0-e61ce88a1c84\" (UID: \"dc5f8d60-4e46-42c2-9de0-e61ce88a1c84\") "
Jul 29 15:08:07.865873 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:07.865718    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2rmq\" (UniqueName: \"kubernetes.io/projected/dc5f8d60-4e46-42c2-9de0-e61ce88a1c84-kube-api-access-w2rmq\") pod \"dc5f8d60-4e46-42c2-9de0-e61ce88a1c84\" (UID: \"dc5f8d60-4e46-42c2-9de0-e61ce88a1c84\") "
Jul 29 15:08:07.866025 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:07.865997    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc5f8d60-4e46-42c2-9de0-e61ce88a1c84-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dc5f8d60-4e46-42c2-9de0-e61ce88a1c84" (UID: "dc5f8d60-4e46-42c2-9de0-e61ce88a1c84"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 15:08:07.867860 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:07.867833    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc5f8d60-4e46-42c2-9de0-e61ce88a1c84-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dc5f8d60-4e46-42c2-9de0-e61ce88a1c84" (UID: "dc5f8d60-4e46-42c2-9de0-e61ce88a1c84"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 15:08:07.867959 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:07.867874    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc5f8d60-4e46-42c2-9de0-e61ce88a1c84-kube-api-access-w2rmq" (OuterVolumeSpecName: "kube-api-access-w2rmq") pod "dc5f8d60-4e46-42c2-9de0-e61ce88a1c84" (UID: "dc5f8d60-4e46-42c2-9de0-e61ce88a1c84"). InnerVolumeSpecName "kube-api-access-w2rmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 15:08:07.966427 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:07.966399    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc5f8d60-4e46-42c2-9de0-e61ce88a1c84-openshift-service-ca\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:08:07.966427 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:07.966426    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc5f8d60-4e46-42c2-9de0-e61ce88a1c84-maas-api-tls\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:08:07.966564 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:07.966436    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2rmq\" (UniqueName: \"kubernetes.io/projected/dc5f8d60-4e46-42c2-9de0-e61ce88a1c84-kube-api-access-w2rmq\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:08:08.432331 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:08.432300    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d27743fd-a4fd-4ffc-923c-8630819dfe5a" path="/var/lib/kubelet/pods/d27743fd-a4fd-4ffc-923c-8630819dfe5a/volumes"
Jul 29 15:08:08.674495 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:08.674463    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0af43e8c-5d4f7bfc96-r824g"
Jul 29 15:08:08.674862 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:08.674467    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0af43e8c-5d4f7bfc96-r824g" event={"ID":"dc5f8d60-4e46-42c2-9de0-e61ce88a1c84","Type":"ContainerDied","Data":"5254de87591230c5607acdf5412d80e300aa07ad581a783c293cac00ec9a7ab0"}
Jul 29 15:08:08.674862 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:08.674627    2563 scope.go:117] "RemoveContainer" containerID="2ce647db3b61172d2ae4abad4d49cd609e44ceaeac6270b565ad0cc17efe94cc"
Jul 29 15:08:08.690926 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:08.690899    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0af43e8c-5d4f7bfc96-r824g"]
Jul 29 15:08:08.693165 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:08.693144    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0af43e8c-5d4f7bfc96-r824g"]
Jul 29 15:08:10.431726 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:10.431692    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc5f8d60-4e46-42c2-9de0-e61ce88a1c84" path="/var/lib/kubelet/pods/dc5f8d60-4e46-42c2-9de0-e61ce88a1c84/volumes"
Jul 29 15:08:14.202987 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:14.202951    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"]
Jul 29 15:08:14.203404 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:14.203350    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x" podUID="9db1f2b9-0437-4cf3-a7d7-fb45b60f181c" containerName="istio-proxy" containerID="cri-o://e4abf47567da98cd46a417b2afae65050ad9dfe58fdd8489c9c444c9deca0a9c" gracePeriod=30
Jul 29 15:08:19.465492 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:19.465466    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"
Jul 29 15:08:19.562620 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:19.562525    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-credential-socket\") pod \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\" (UID: \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\") "
Jul 29 15:08:19.562620 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:19.562575    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-istio-envoy\") pod \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\" (UID: \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\") "
Jul 29 15:08:19.562620 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:19.562598    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-istiod-ca-cert\") pod \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\" (UID: \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\") "
Jul 29 15:08:19.562881 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:19.562629    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-istio-token\") pod \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\" (UID: \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\") "
Jul 29 15:08:19.562881 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:19.562658    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-istio-podinfo\") pod \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\" (UID: \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\") "
Jul 29 15:08:19.562881 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:19.562699    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-workload-certs\") pod \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\" (UID: \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\") "
Jul 29 15:08:19.562881 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:19.562796    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-workload-socket\") pod \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\" (UID: \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\") "
Jul 29 15:08:19.562881 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:19.562860    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jf7wf\" (UniqueName: \"kubernetes.io/projected/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-kube-api-access-jf7wf\") pod \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\" (UID: \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\") "
Jul 29 15:08:19.563157 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:19.562918    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-istio-data\") pod \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\" (UID: \"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c\") "
Jul 29 15:08:19.563157 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:19.562933    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9db1f2b9-0437-4cf3-a7d7-fb45b60f181c" (UID: "9db1f2b9-0437-4cf3-a7d7-fb45b60f181c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:08:19.563157 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:19.563026    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9db1f2b9-0437-4cf3-a7d7-fb45b60f181c" (UID: "9db1f2b9-0437-4cf3-a7d7-fb45b60f181c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:08:19.563350 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:19.563125    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9db1f2b9-0437-4cf3-a7d7-fb45b60f181c" (UID: "9db1f2b9-0437-4cf3-a7d7-fb45b60f181c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:08:19.563350 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:19.563202    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9db1f2b9-0437-4cf3-a7d7-fb45b60f181c" (UID: "9db1f2b9-0437-4cf3-a7d7-fb45b60f181c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 15:08:19.563350 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:19.563298    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-workload-certs\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:08:19.563350 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:19.563318    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-workload-socket\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:08:19.563350 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:19.563332    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-credential-socket\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:08:19.563350 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:19.563345    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-istiod-ca-cert\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:08:19.563901 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:19.563880    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-istio-data" (OuterVolumeSpecName: "istio-data") pod "9db1f2b9-0437-4cf3-a7d7-fb45b60f181c" (UID: "9db1f2b9-0437-4cf3-a7d7-fb45b60f181c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:08:19.564854 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:19.564832    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9db1f2b9-0437-4cf3-a7d7-fb45b60f181c" (UID: "9db1f2b9-0437-4cf3-a7d7-fb45b60f181c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:08:19.564980 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:19.564963    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-istio-token" (OuterVolumeSpecName: "istio-token") pod "9db1f2b9-0437-4cf3-a7d7-fb45b60f181c" (UID: "9db1f2b9-0437-4cf3-a7d7-fb45b60f181c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 15:08:19.565312 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:19.565289    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9db1f2b9-0437-4cf3-a7d7-fb45b60f181c" (UID: "9db1f2b9-0437-4cf3-a7d7-fb45b60f181c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 15:08:19.565312 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:19.565305    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-kube-api-access-jf7wf" (OuterVolumeSpecName: "kube-api-access-jf7wf") pod "9db1f2b9-0437-4cf3-a7d7-fb45b60f181c" (UID: "9db1f2b9-0437-4cf3-a7d7-fb45b60f181c"). InnerVolumeSpecName "kube-api-access-jf7wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 15:08:19.663938 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:19.663901    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-istio-envoy\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:08:19.663938 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:19.663930    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-istio-token\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:08:19.663938 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:19.663939    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-istio-podinfo\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:08:19.664185 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:19.663948    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf7wf\" (UniqueName: \"kubernetes.io/projected/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-kube-api-access-jf7wf\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:08:19.664185 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:19.663958    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c-istio-data\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:08:19.728398 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:19.728367    2563 generic.go:356] "Generic (PLEG): container finished" podID="9db1f2b9-0437-4cf3-a7d7-fb45b60f181c" containerID="e4abf47567da98cd46a417b2afae65050ad9dfe58fdd8489c9c444c9deca0a9c" exitCode=0
Jul 29 15:08:19.728558 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:19.728426    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"
Jul 29 15:08:19.728558 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:19.728458    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x" event={"ID":"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c","Type":"ContainerDied","Data":"e4abf47567da98cd46a417b2afae65050ad9dfe58fdd8489c9c444c9deca0a9c"}
Jul 29 15:08:19.728558 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:19.728550    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x" event={"ID":"9db1f2b9-0437-4cf3-a7d7-fb45b60f181c","Type":"ContainerDied","Data":"75a25ccf2ae8cf1d27439effa9cdae455f52b5d1f97d26f7f71f473647edd253"}
Jul 29 15:08:19.728732 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:19.728581    2563 scope.go:117] "RemoveContainer" containerID="e4abf47567da98cd46a417b2afae65050ad9dfe58fdd8489c9c444c9deca0a9c"
Jul 29 15:08:19.737644 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:19.737629    2563 scope.go:117] "RemoveContainer" containerID="7976f9c0d563ab8ff5811681267471a3e18e7bea84ce1572acd675f9f59c533c"
Jul 29 15:08:19.745663 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:19.745646    2563 scope.go:117] "RemoveContainer" containerID="e4abf47567da98cd46a417b2afae65050ad9dfe58fdd8489c9c444c9deca0a9c"
Jul 29 15:08:19.745895 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:08:19.745874    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4abf47567da98cd46a417b2afae65050ad9dfe58fdd8489c9c444c9deca0a9c\": container with ID starting with e4abf47567da98cd46a417b2afae65050ad9dfe58fdd8489c9c444c9deca0a9c not found: ID does not exist" containerID="e4abf47567da98cd46a417b2afae65050ad9dfe58fdd8489c9c444c9deca0a9c"
Jul 29 15:08:19.745967 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:19.745907    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4abf47567da98cd46a417b2afae65050ad9dfe58fdd8489c9c444c9deca0a9c"} err="failed to get container status \"e4abf47567da98cd46a417b2afae65050ad9dfe58fdd8489c9c444c9deca0a9c\": rpc error: code = NotFound desc = could not find container \"e4abf47567da98cd46a417b2afae65050ad9dfe58fdd8489c9c444c9deca0a9c\": container with ID starting with e4abf47567da98cd46a417b2afae65050ad9dfe58fdd8489c9c444c9deca0a9c not found: ID does not exist"
Jul 29 15:08:19.745967 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:19.745931    2563 scope.go:117] "RemoveContainer" containerID="7976f9c0d563ab8ff5811681267471a3e18e7bea84ce1572acd675f9f59c533c"
Jul 29 15:08:19.746208 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:08:19.746191    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7976f9c0d563ab8ff5811681267471a3e18e7bea84ce1572acd675f9f59c533c\": container with ID starting with 7976f9c0d563ab8ff5811681267471a3e18e7bea84ce1572acd675f9f59c533c not found: ID does not exist" containerID="7976f9c0d563ab8ff5811681267471a3e18e7bea84ce1572acd675f9f59c533c"
Jul 29 15:08:19.746254 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:19.746215    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7976f9c0d563ab8ff5811681267471a3e18e7bea84ce1572acd675f9f59c533c"} err="failed to get container status \"7976f9c0d563ab8ff5811681267471a3e18e7bea84ce1572acd675f9f59c533c\": rpc error: code = NotFound desc = could not find container \"7976f9c0d563ab8ff5811681267471a3e18e7bea84ce1572acd675f9f59c533c\": container with ID starting with 7976f9c0d563ab8ff5811681267471a3e18e7bea84ce1572acd675f9f59c533c not found: ID does not exist"
Jul 29 15:08:19.749948 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:19.749927    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"]
Jul 29 15:08:19.754579 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:19.754557    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-0af43e8c-openshift-default-577c66768d-bqj2x"]
Jul 29 15:08:20.432444 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:20.432411    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9db1f2b9-0437-4cf3-a7d7-fb45b60f181c" path="/var/lib/kubelet/pods/9db1f2b9-0437-4cf3-a7d7-fb45b60f181c/volumes"
Jul 29 15:08:34.185517 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.185483    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"]
Jul 29 15:08:34.186007 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.185851    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d27743fd-a4fd-4ffc-923c-8630819dfe5a" containerName="revoke-keys"
Jul 29 15:08:34.186007 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.185862    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d27743fd-a4fd-4ffc-923c-8630819dfe5a" containerName="revoke-keys"
Jul 29 15:08:34.186007 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.185881    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc5f8d60-4e46-42c2-9de0-e61ce88a1c84" containerName="maas-api"
Jul 29 15:08:34.186007 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.185886    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc5f8d60-4e46-42c2-9de0-e61ce88a1c84" containerName="maas-api"
Jul 29 15:08:34.186007 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.185894    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9db1f2b9-0437-4cf3-a7d7-fb45b60f181c" containerName="istio-proxy"
Jul 29 15:08:34.186007 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.185899    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9db1f2b9-0437-4cf3-a7d7-fb45b60f181c" containerName="istio-proxy"
Jul 29 15:08:34.186007 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.185922    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9db1f2b9-0437-4cf3-a7d7-fb45b60f181c" containerName="istio-proxy"
Jul 29 15:08:34.186007 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.185928    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9db1f2b9-0437-4cf3-a7d7-fb45b60f181c" containerName="istio-proxy"
Jul 29 15:08:34.186007 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.185981    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc5f8d60-4e46-42c2-9de0-e61ce88a1c84" containerName="maas-api"
Jul 29 15:08:34.186007 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.185990    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9db1f2b9-0437-4cf3-a7d7-fb45b60f181c" containerName="istio-proxy"
Jul 29 15:08:34.186007 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.186000    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d27743fd-a4fd-4ffc-923c-8630819dfe5a" containerName="revoke-keys"
Jul 29 15:08:34.189009 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.188993    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"
Jul 29 15:08:34.191279 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.191254    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 29 15:08:34.191403 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.191297    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a385d8ab-openshift-default-dockercfg-fgtzt\""
Jul 29 15:08:34.208318 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.208295    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"]
Jul 29 15:08:34.281417 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.281383    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0847c0bd-0e85-447f-99c0-81552162797d-istio-envoy\") pod \"e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2\" (UID: \"0847c0bd-0e85-447f-99c0-81552162797d\") " pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"
Jul 29 15:08:34.281417 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.281419    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0847c0bd-0e85-447f-99c0-81552162797d-workload-certs\") pod \"e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2\" (UID: \"0847c0bd-0e85-447f-99c0-81552162797d\") " pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"
Jul 29 15:08:34.281625 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.281437    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0847c0bd-0e85-447f-99c0-81552162797d-istio-data\") pod \"e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2\" (UID: \"0847c0bd-0e85-447f-99c0-81552162797d\") " pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"
Jul 29 15:08:34.281625 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.281458    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c65s\" (UniqueName: \"kubernetes.io/projected/0847c0bd-0e85-447f-99c0-81552162797d-kube-api-access-5c65s\") pod \"e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2\" (UID: \"0847c0bd-0e85-447f-99c0-81552162797d\") " pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"
Jul 29 15:08:34.281625 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.281505    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0847c0bd-0e85-447f-99c0-81552162797d-workload-socket\") pod \"e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2\" (UID: \"0847c0bd-0e85-447f-99c0-81552162797d\") " pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"
Jul 29 15:08:34.281625 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.281520    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0847c0bd-0e85-447f-99c0-81552162797d-credential-socket\") pod \"e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2\" (UID: \"0847c0bd-0e85-447f-99c0-81552162797d\") " pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"
Jul 29 15:08:34.281625 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.281579    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0847c0bd-0e85-447f-99c0-81552162797d-istio-token\") pod \"e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2\" (UID: \"0847c0bd-0e85-447f-99c0-81552162797d\") " pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"
Jul 29 15:08:34.281816 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.281629    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0847c0bd-0e85-447f-99c0-81552162797d-istio-podinfo\") pod \"e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2\" (UID: \"0847c0bd-0e85-447f-99c0-81552162797d\") " pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"
Jul 29 15:08:34.281816 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.281658    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0847c0bd-0e85-447f-99c0-81552162797d-istiod-ca-cert\") pod \"e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2\" (UID: \"0847c0bd-0e85-447f-99c0-81552162797d\") " pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"
Jul 29 15:08:34.382708 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.382671    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0847c0bd-0e85-447f-99c0-81552162797d-istio-envoy\") pod \"e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2\" (UID: \"0847c0bd-0e85-447f-99c0-81552162797d\") " pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"
Jul 29 15:08:34.382905 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.382718    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0847c0bd-0e85-447f-99c0-81552162797d-workload-certs\") pod \"e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2\" (UID: \"0847c0bd-0e85-447f-99c0-81552162797d\") " pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"
Jul 29 15:08:34.382905 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.382744    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0847c0bd-0e85-447f-99c0-81552162797d-istio-data\") pod \"e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2\" (UID: \"0847c0bd-0e85-447f-99c0-81552162797d\") " pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"
Jul 29 15:08:34.382905 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.382771    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c65s\" (UniqueName: \"kubernetes.io/projected/0847c0bd-0e85-447f-99c0-81552162797d-kube-api-access-5c65s\") pod \"e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2\" (UID: \"0847c0bd-0e85-447f-99c0-81552162797d\") " pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"
Jul 29 15:08:34.382905 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.382816    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0847c0bd-0e85-447f-99c0-81552162797d-workload-socket\") pod \"e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2\" (UID: \"0847c0bd-0e85-447f-99c0-81552162797d\") " pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"
Jul 29 15:08:34.383195 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.382973    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0847c0bd-0e85-447f-99c0-81552162797d-credential-socket\") pod \"e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2\" (UID: \"0847c0bd-0e85-447f-99c0-81552162797d\") " pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"
Jul 29 15:08:34.383195 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.383038    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0847c0bd-0e85-447f-99c0-81552162797d-istio-token\") pod \"e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2\" (UID: \"0847c0bd-0e85-447f-99c0-81552162797d\") " pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"
Jul 29 15:08:34.383195 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.383106    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0847c0bd-0e85-447f-99c0-81552162797d-istio-podinfo\") pod \"e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2\" (UID: \"0847c0bd-0e85-447f-99c0-81552162797d\") " pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"
Jul 29 15:08:34.383195 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.383180    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0847c0bd-0e85-447f-99c0-81552162797d-istiod-ca-cert\") pod \"e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2\" (UID: \"0847c0bd-0e85-447f-99c0-81552162797d\") " pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"
Jul 29 15:08:34.383195 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.383181    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0847c0bd-0e85-447f-99c0-81552162797d-workload-certs\") pod \"e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2\" (UID: \"0847c0bd-0e85-447f-99c0-81552162797d\") " pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"
Jul 29 15:08:34.383463 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.383442    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0847c0bd-0e85-447f-99c0-81552162797d-istio-data\") pod \"e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2\" (UID: \"0847c0bd-0e85-447f-99c0-81552162797d\") " pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"
Jul 29 15:08:34.383534 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.383519    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0847c0bd-0e85-447f-99c0-81552162797d-credential-socket\") pod \"e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2\" (UID: \"0847c0bd-0e85-447f-99c0-81552162797d\") " pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"
Jul 29 15:08:34.383905 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.383884    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0847c0bd-0e85-447f-99c0-81552162797d-workload-socket\") pod \"e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2\" (UID: \"0847c0bd-0e85-447f-99c0-81552162797d\") " pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"
Jul 29 15:08:34.383969 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.383942    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0847c0bd-0e85-447f-99c0-81552162797d-istiod-ca-cert\") pod \"e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2\" (UID: \"0847c0bd-0e85-447f-99c0-81552162797d\") " pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"
Jul 29 15:08:34.385278 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.385254    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0847c0bd-0e85-447f-99c0-81552162797d-istio-envoy\") pod \"e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2\" (UID: \"0847c0bd-0e85-447f-99c0-81552162797d\") " pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"
Jul 29 15:08:34.385550 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.385532    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0847c0bd-0e85-447f-99c0-81552162797d-istio-podinfo\") pod \"e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2\" (UID: \"0847c0bd-0e85-447f-99c0-81552162797d\") " pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"
Jul 29 15:08:34.393537 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.393519    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0847c0bd-0e85-447f-99c0-81552162797d-istio-token\") pod \"e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2\" (UID: \"0847c0bd-0e85-447f-99c0-81552162797d\") " pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"
Jul 29 15:08:34.393740 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.393709    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c65s\" (UniqueName: \"kubernetes.io/projected/0847c0bd-0e85-447f-99c0-81552162797d-kube-api-access-5c65s\") pod \"e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2\" (UID: \"0847c0bd-0e85-447f-99c0-81552162797d\") " pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"
Jul 29 15:08:34.499489 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.499466    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"
Jul 29 15:08:34.640837 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.640811    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"]
Jul 29 15:08:34.642645 ip-10-0-136-196 kubenswrapper[2563]: W0729 15:08:34.642615    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0847c0bd_0e85_447f_99c0_81552162797d.slice/crio-5f33bd0fba0ed8fa3b21eb7afba1cef1fc3ce4fde06aac539dd07dff58c5bfd1 WatchSource:0}: Error finding container 5f33bd0fba0ed8fa3b21eb7afba1cef1fc3ce4fde06aac539dd07dff58c5bfd1: Status 404 returned error can't find the container with id 5f33bd0fba0ed8fa3b21eb7afba1cef1fc3ce4fde06aac539dd07dff58c5bfd1
Jul 29 15:08:34.645373 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.645337    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:08:34.645453 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.645418    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:08:34.645496 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.645485    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:08:34.807192 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.807072    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2" event={"ID":"0847c0bd-0e85-447f-99c0-81552162797d","Type":"ContainerStarted","Data":"bc3d19f0c647f0c92a5ceffdaa5dd58f132d04ccdf7835b296e7115c0c176cb5"}
Jul 29 15:08:34.807192 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.807152    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2" event={"ID":"0847c0bd-0e85-447f-99c0-81552162797d","Type":"ContainerStarted","Data":"5f33bd0fba0ed8fa3b21eb7afba1cef1fc3ce4fde06aac539dd07dff58c5bfd1"}
Jul 29 15:08:34.827661 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:34.827594    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2" podStartSLOduration=0.827573652 podStartE2EDuration="827.573652ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:08:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:08:34.826202954 +0000 UTC m=+2016.874660895" watchObservedRunningTime="2026-07-29 15:08:34.827573652 +0000 UTC m=+2016.876031590"
Jul 29 15:08:35.501780 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:35.501738    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"
Jul 29 15:08:35.506906 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:35.506876    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"
Jul 29 15:08:35.813200 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:35.813101    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"
Jul 29 15:08:35.813970 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:35.813948    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"
Jul 29 15:08:37.000754 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:37.000717    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a385d8ab-f48b9d6cb-vcnzw"]
Jul 29 15:08:37.001210 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:37.001195    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9db1f2b9-0437-4cf3-a7d7-fb45b60f181c" containerName="istio-proxy"
Jul 29 15:08:37.004424 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:37.004403    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a385d8ab-f48b9d6cb-vcnzw"
Jul 29 15:08:37.004820 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:37.004793    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3d287fd-cc0b-47a9-a3b6-86aada844176-openshift-service-ca\") pod \"maas-api-e2e-mt-a385d8ab-f48b9d6cb-vcnzw\" (UID: \"e3d287fd-cc0b-47a9-a3b6-86aada844176\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a385d8ab-f48b9d6cb-vcnzw"
Jul 29 15:08:37.004897 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:37.004834    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdnj2\" (UniqueName: \"kubernetes.io/projected/e3d287fd-cc0b-47a9-a3b6-86aada844176-kube-api-access-fdnj2\") pod \"maas-api-e2e-mt-a385d8ab-f48b9d6cb-vcnzw\" (UID: \"e3d287fd-cc0b-47a9-a3b6-86aada844176\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a385d8ab-f48b9d6cb-vcnzw"
Jul 29 15:08:37.004981 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:37.004957    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3d287fd-cc0b-47a9-a3b6-86aada844176-maas-api-tls\") pod \"maas-api-e2e-mt-a385d8ab-f48b9d6cb-vcnzw\" (UID: \"e3d287fd-cc0b-47a9-a3b6-86aada844176\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a385d8ab-f48b9d6cb-vcnzw"
Jul 29 15:08:37.006968 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:37.006950    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a385d8ab-serving-cert\""
Jul 29 15:08:37.017432 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:37.017403    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a385d8ab-f48b9d6cb-vcnzw"]
Jul 29 15:08:37.106216 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:37.106180    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3d287fd-cc0b-47a9-a3b6-86aada844176-openshift-service-ca\") pod \"maas-api-e2e-mt-a385d8ab-f48b9d6cb-vcnzw\" (UID: \"e3d287fd-cc0b-47a9-a3b6-86aada844176\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a385d8ab-f48b9d6cb-vcnzw"
Jul 29 15:08:37.106216 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:37.106217    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdnj2\" (UniqueName: \"kubernetes.io/projected/e3d287fd-cc0b-47a9-a3b6-86aada844176-kube-api-access-fdnj2\") pod \"maas-api-e2e-mt-a385d8ab-f48b9d6cb-vcnzw\" (UID: \"e3d287fd-cc0b-47a9-a3b6-86aada844176\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a385d8ab-f48b9d6cb-vcnzw"
Jul 29 15:08:37.106442 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:37.106272    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3d287fd-cc0b-47a9-a3b6-86aada844176-maas-api-tls\") pod \"maas-api-e2e-mt-a385d8ab-f48b9d6cb-vcnzw\" (UID: \"e3d287fd-cc0b-47a9-a3b6-86aada844176\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a385d8ab-f48b9d6cb-vcnzw"
Jul 29 15:08:37.106442 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:08:37.106420    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-a385d8ab-serving-cert: secret "maas-api-e2e-mt-a385d8ab-serving-cert" not found
Jul 29 15:08:37.106525 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:08:37.106482    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3d287fd-cc0b-47a9-a3b6-86aada844176-maas-api-tls podName:e3d287fd-cc0b-47a9-a3b6-86aada844176 nodeName:}" failed. No retries permitted until 2026-07-29 15:08:37.606461566 +0000 UTC m=+2019.654919489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e3d287fd-cc0b-47a9-a3b6-86aada844176-maas-api-tls") pod "maas-api-e2e-mt-a385d8ab-f48b9d6cb-vcnzw" (UID: "e3d287fd-cc0b-47a9-a3b6-86aada844176") : secret "maas-api-e2e-mt-a385d8ab-serving-cert" not found
Jul 29 15:08:37.106938 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:37.106918    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3d287fd-cc0b-47a9-a3b6-86aada844176-openshift-service-ca\") pod \"maas-api-e2e-mt-a385d8ab-f48b9d6cb-vcnzw\" (UID: \"e3d287fd-cc0b-47a9-a3b6-86aada844176\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a385d8ab-f48b9d6cb-vcnzw"
Jul 29 15:08:37.127088 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:37.127058    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdnj2\" (UniqueName: \"kubernetes.io/projected/e3d287fd-cc0b-47a9-a3b6-86aada844176-kube-api-access-fdnj2\") pod \"maas-api-e2e-mt-a385d8ab-f48b9d6cb-vcnzw\" (UID: \"e3d287fd-cc0b-47a9-a3b6-86aada844176\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a385d8ab-f48b9d6cb-vcnzw"
Jul 29 15:08:37.609658 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:37.609623    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3d287fd-cc0b-47a9-a3b6-86aada844176-maas-api-tls\") pod \"maas-api-e2e-mt-a385d8ab-f48b9d6cb-vcnzw\" (UID: \"e3d287fd-cc0b-47a9-a3b6-86aada844176\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a385d8ab-f48b9d6cb-vcnzw"
Jul 29 15:08:37.612124 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:37.612091    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3d287fd-cc0b-47a9-a3b6-86aada844176-maas-api-tls\") pod \"maas-api-e2e-mt-a385d8ab-f48b9d6cb-vcnzw\" (UID: \"e3d287fd-cc0b-47a9-a3b6-86aada844176\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a385d8ab-f48b9d6cb-vcnzw"
Jul 29 15:08:37.615911 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:37.615889    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a385d8ab-f48b9d6cb-vcnzw"
Jul 29 15:08:37.765994 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:37.765973    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a385d8ab-f48b9d6cb-vcnzw"]
Jul 29 15:08:37.768096 ip-10-0-136-196 kubenswrapper[2563]: W0729 15:08:37.768069    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3d287fd_cc0b_47a9_a3b6_86aada844176.slice/crio-7fedd2949677f2484e05e4bb1f652b30e9cbd1976059ccab3261df3361c99367 WatchSource:0}: Error finding container 7fedd2949677f2484e05e4bb1f652b30e9cbd1976059ccab3261df3361c99367: Status 404 returned error can't find the container with id 7fedd2949677f2484e05e4bb1f652b30e9cbd1976059ccab3261df3361c99367
Jul 29 15:08:37.769667 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:37.769643    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 15:08:37.823468 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:37.823437    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a385d8ab-f48b9d6cb-vcnzw" event={"ID":"e3d287fd-cc0b-47a9-a3b6-86aada844176","Type":"ContainerStarted","Data":"7fedd2949677f2484e05e4bb1f652b30e9cbd1976059ccab3261df3361c99367"}
Jul 29 15:08:38.829969 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:38.829927    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a385d8ab-f48b9d6cb-vcnzw" event={"ID":"e3d287fd-cc0b-47a9-a3b6-86aada844176","Type":"ContainerStarted","Data":"228c116ef10727eb1269278b06b165a07deef9f9c6241dc56e151e400b4582b6"}
Jul 29 15:08:38.830383 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:38.830005    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a385d8ab-f48b9d6cb-vcnzw"
Jul 29 15:08:38.856599 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:38.856549    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a385d8ab-f48b9d6cb-vcnzw" podStartSLOduration=2.335536971 podStartE2EDuration="2.856534383s" totalImagesPullingTime="520.997412ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:08:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:08:38.851854651 +0000 UTC m=+2020.900312586" watchObservedRunningTime="2026-07-29 15:08:38.856534383 +0000 UTC m=+2020.904992325"
Jul 29 15:08:44.841508 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:44.841473    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a385d8ab-f48b9d6cb-vcnzw"
Jul 29 15:08:51.436457 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:51.435889    2563 patch_prober.go:28] interesting pod/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": read tcp 10.132.0.2:36000->10.132.0.58:15021: read: connection reset by peer" start-of-body=
Jul 29 15:08:51.436457 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:51.435967    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2" podUID="0847c0bd-0e85-447f-99c0-81552162797d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": read tcp 10.132.0.2:36000->10.132.0.58:15021: read: connection reset by peer"
Jul 29 15:08:51.905603 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:51.905570    2563 generic.go:356] "Generic (PLEG): container finished" podID="0847c0bd-0e85-447f-99c0-81552162797d" containerID="bc3d19f0c647f0c92a5ceffdaa5dd58f132d04ccdf7835b296e7115c0c176cb5" exitCode=0
Jul 29 15:08:51.905603 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:51.905609    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2" event={"ID":"0847c0bd-0e85-447f-99c0-81552162797d","Type":"ContainerDied","Data":"bc3d19f0c647f0c92a5ceffdaa5dd58f132d04ccdf7835b296e7115c0c176cb5"}
Jul 29 15:08:51.905981 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:51.905965    2563 scope.go:117] "RemoveContainer" containerID="bc3d19f0c647f0c92a5ceffdaa5dd58f132d04ccdf7835b296e7115c0c176cb5"
Jul 29 15:08:51.906953 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:51.906926    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:08:51.907018 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:51.906994    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:08:51.907052 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:51.907026    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:08:52.912201 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:52.912166    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2" event={"ID":"0847c0bd-0e85-447f-99c0-81552162797d","Type":"ContainerStarted","Data":"4a540a79a233b10e4033565f5ad60fea6f9a50dd42c882e4beb2197606f4806d"}
Jul 29 15:08:53.500201 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:53.500156    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"
Jul 29 15:08:53.500471 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:53.500448    2563 patch_prober.go:28] interesting pod/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Jul 29 15:08:53.500518 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:53.500499    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2" podUID="0847c0bd-0e85-447f-99c0-81552162797d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Jul 29 15:08:54.500887 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:54.500850    2563 patch_prober.go:28] interesting pod/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Jul 29 15:08:54.501347 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:54.500903    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2" podUID="0847c0bd-0e85-447f-99c0-81552162797d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Jul 29 15:08:55.501059 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:55.501029    2563 patch_prober.go:28] interesting pod/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Jul 29 15:08:55.501455 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:55.501089    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2" podUID="0847c0bd-0e85-447f-99c0-81552162797d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Jul 29 15:08:56.500201 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:56.500157    2563 patch_prober.go:28] interesting pod/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Jul 29 15:08:56.500371 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:56.500221    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2" podUID="0847c0bd-0e85-447f-99c0-81552162797d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Jul 29 15:08:57.499933 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:57.499876    2563 patch_prober.go:28] interesting pod/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Jul 29 15:08:57.499933 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:57.499924    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2" podUID="0847c0bd-0e85-447f-99c0-81552162797d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Jul 29 15:08:58.503628 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:58.503602    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"
Jul 29 15:08:58.503973 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:58.503851    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"
Jul 29 15:08:58.504448 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:08:58.504431    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"
Jul 29 15:09:58.810220 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:09:58.810176    2563 scope.go:117] "RemoveContainer" containerID="540bf37ef99504ebd4bb29d422930704ff97c30475a566165fb87bddec98a90f"
Jul 29 15:09:58.812883 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:09:58.812857    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xnm95_23d79588-937d-4c6d-8760-a081024f608e/console-operator/1.log"
Jul 29 15:09:58.813594 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:09:58.813572    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xnm95_23d79588-937d-4c6d-8760-a081024f608e/console-operator/1.log"
Jul 29 15:10:02.182879 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:02.182842    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a385d8ab-4bg6g"]
Jul 29 15:10:02.187722 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:02.187701    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a385d8ab-4bg6g"
Jul 29 15:10:02.190249 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:02.190222    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mmt6p\""
Jul 29 15:10:02.195479 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:02.195453    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a385d8ab-4bg6g"]
Jul 29 15:10:02.367870 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:02.367832    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf3ea83-3a47-4c8a-ad54-7dde8d82a41e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a385d8ab-4bg6g\" (UID: \"fdf3ea83-3a47-4c8a-ad54-7dde8d82a41e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a385d8ab-4bg6g"
Jul 29 15:10:02.468746 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:02.468712    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf3ea83-3a47-4c8a-ad54-7dde8d82a41e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a385d8ab-4bg6g\" (UID: \"fdf3ea83-3a47-4c8a-ad54-7dde8d82a41e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a385d8ab-4bg6g"
Jul 29 15:10:02.469372 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:02.469352    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf3ea83-3a47-4c8a-ad54-7dde8d82a41e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a385d8ab-4bg6g\" (UID: \"fdf3ea83-3a47-4c8a-ad54-7dde8d82a41e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a385d8ab-4bg6g"
Jul 29 15:10:02.504890 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:02.504863    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a385d8ab-4bg6g"
Jul 29 15:10:02.637175 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:02.637120    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a385d8ab-4bg6g"]
Jul 29 15:10:02.638857 ip-10-0-136-196 kubenswrapper[2563]: W0729 15:10:02.638834    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf3ea83_3a47_4c8a_ad54_7dde8d82a41e.slice/crio-e415f7f67009bd64694760a60bd862de74abdd7994d16d451283fb545346c7c7 WatchSource:0}: Error finding container e415f7f67009bd64694760a60bd862de74abdd7994d16d451283fb545346c7c7: Status 404 returned error can't find the container with id e415f7f67009bd64694760a60bd862de74abdd7994d16d451283fb545346c7c7
Jul 29 15:10:03.266970 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:03.266937    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a385d8ab-4bg6g" event={"ID":"fdf3ea83-3a47-4c8a-ad54-7dde8d82a41e","Type":"ContainerStarted","Data":"a812e61ff3a85d566436b396252ba096b5e884f2bb524f5b2535b4a7ac338242"}
Jul 29 15:10:03.267403 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:03.266991    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a385d8ab-4bg6g" event={"ID":"fdf3ea83-3a47-4c8a-ad54-7dde8d82a41e","Type":"ContainerStarted","Data":"e415f7f67009bd64694760a60bd862de74abdd7994d16d451283fb545346c7c7"}
Jul 29 15:10:03.282485 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:03.282437    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a385d8ab-4bg6g" podStartSLOduration=1.282420112 podStartE2EDuration="1.282420112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:10:03.281324386 +0000 UTC m=+2105.329782325" watchObservedRunningTime="2026-07-29 15:10:03.282420112 +0000 UTC m=+2105.330878051"
Jul 29 15:10:04.273773 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:04.273739    2563 generic.go:356] "Generic (PLEG): container finished" podID="fdf3ea83-3a47-4c8a-ad54-7dde8d82a41e" containerID="a812e61ff3a85d566436b396252ba096b5e884f2bb524f5b2535b4a7ac338242" exitCode=0
Jul 29 15:10:04.274232 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:04.273828    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a385d8ab-4bg6g" event={"ID":"fdf3ea83-3a47-4c8a-ad54-7dde8d82a41e","Type":"ContainerDied","Data":"a812e61ff3a85d566436b396252ba096b5e884f2bb524f5b2535b4a7ac338242"}
Jul 29 15:10:05.418191 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:05.418170    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a385d8ab-4bg6g"
Jul 29 15:10:05.596794 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:05.596710    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf3ea83-3a47-4c8a-ad54-7dde8d82a41e-maas-api-service-ca\") pod \"fdf3ea83-3a47-4c8a-ad54-7dde8d82a41e\" (UID: \"fdf3ea83-3a47-4c8a-ad54-7dde8d82a41e\") "
Jul 29 15:10:05.597054 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:05.597030    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdf3ea83-3a47-4c8a-ad54-7dde8d82a41e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fdf3ea83-3a47-4c8a-ad54-7dde8d82a41e" (UID: "fdf3ea83-3a47-4c8a-ad54-7dde8d82a41e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 15:10:05.697547 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:05.697499    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf3ea83-3a47-4c8a-ad54-7dde8d82a41e-maas-api-service-ca\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:10:06.285789 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:06.285761    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a385d8ab-4bg6g"
Jul 29 15:10:06.285789 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:06.285773    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a385d8ab-4bg6g" event={"ID":"fdf3ea83-3a47-4c8a-ad54-7dde8d82a41e","Type":"ContainerDied","Data":"e415f7f67009bd64694760a60bd862de74abdd7994d16d451283fb545346c7c7"}
Jul 29 15:10:06.286001 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:06.285822    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e415f7f67009bd64694760a60bd862de74abdd7994d16d451283fb545346c7c7"
Jul 29 15:10:17.865820 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:17.865783    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a385d8ab-f48b9d6cb-vcnzw"]
Jul 29 15:10:17.866417 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:17.866061    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a385d8ab-f48b9d6cb-vcnzw" podUID="e3d287fd-cc0b-47a9-a3b6-86aada844176" containerName="maas-api" containerID="cri-o://228c116ef10727eb1269278b06b165a07deef9f9c6241dc56e151e400b4582b6" gracePeriod=30
Jul 29 15:10:18.094630 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:18.094600    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a385d8ab-4bg6g"]
Jul 29 15:10:18.097692 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:18.097666    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a385d8ab-4bg6g"]
Jul 29 15:10:18.124179 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:18.124106    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a385d8ab-f48b9d6cb-vcnzw"
Jul 29 15:10:18.204336 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:18.204301    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdnj2\" (UniqueName: \"kubernetes.io/projected/e3d287fd-cc0b-47a9-a3b6-86aada844176-kube-api-access-fdnj2\") pod \"e3d287fd-cc0b-47a9-a3b6-86aada844176\" (UID: \"e3d287fd-cc0b-47a9-a3b6-86aada844176\") "
Jul 29 15:10:18.204536 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:18.204380    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3d287fd-cc0b-47a9-a3b6-86aada844176-maas-api-tls\") pod \"e3d287fd-cc0b-47a9-a3b6-86aada844176\" (UID: \"e3d287fd-cc0b-47a9-a3b6-86aada844176\") "
Jul 29 15:10:18.204536 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:18.204448    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3d287fd-cc0b-47a9-a3b6-86aada844176-openshift-service-ca\") pod \"e3d287fd-cc0b-47a9-a3b6-86aada844176\" (UID: \"e3d287fd-cc0b-47a9-a3b6-86aada844176\") "
Jul 29 15:10:18.204750 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:18.204725    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3d287fd-cc0b-47a9-a3b6-86aada844176-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e3d287fd-cc0b-47a9-a3b6-86aada844176" (UID: "e3d287fd-cc0b-47a9-a3b6-86aada844176"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 15:10:18.206473 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:18.206444    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3d287fd-cc0b-47a9-a3b6-86aada844176-kube-api-access-fdnj2" (OuterVolumeSpecName: "kube-api-access-fdnj2") pod "e3d287fd-cc0b-47a9-a3b6-86aada844176" (UID: "e3d287fd-cc0b-47a9-a3b6-86aada844176"). InnerVolumeSpecName "kube-api-access-fdnj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 15:10:18.206473 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:18.206448    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3d287fd-cc0b-47a9-a3b6-86aada844176-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e3d287fd-cc0b-47a9-a3b6-86aada844176" (UID: "e3d287fd-cc0b-47a9-a3b6-86aada844176"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 15:10:18.305384 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:18.305355    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdnj2\" (UniqueName: \"kubernetes.io/projected/e3d287fd-cc0b-47a9-a3b6-86aada844176-kube-api-access-fdnj2\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:10:18.305384 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:18.305379    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3d287fd-cc0b-47a9-a3b6-86aada844176-maas-api-tls\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:10:18.305384 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:18.305389    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3d287fd-cc0b-47a9-a3b6-86aada844176-openshift-service-ca\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:10:18.343765 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:18.343738    2563 generic.go:356] "Generic (PLEG): container finished" podID="e3d287fd-cc0b-47a9-a3b6-86aada844176" containerID="228c116ef10727eb1269278b06b165a07deef9f9c6241dc56e151e400b4582b6" exitCode=0
Jul 29 15:10:18.343890 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:18.343785    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a385d8ab-f48b9d6cb-vcnzw"
Jul 29 15:10:18.343890 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:18.343806    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a385d8ab-f48b9d6cb-vcnzw" event={"ID":"e3d287fd-cc0b-47a9-a3b6-86aada844176","Type":"ContainerDied","Data":"228c116ef10727eb1269278b06b165a07deef9f9c6241dc56e151e400b4582b6"}
Jul 29 15:10:18.343890 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:18.343846    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a385d8ab-f48b9d6cb-vcnzw" event={"ID":"e3d287fd-cc0b-47a9-a3b6-86aada844176","Type":"ContainerDied","Data":"7fedd2949677f2484e05e4bb1f652b30e9cbd1976059ccab3261df3361c99367"}
Jul 29 15:10:18.343890 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:18.343868    2563 scope.go:117] "RemoveContainer" containerID="228c116ef10727eb1269278b06b165a07deef9f9c6241dc56e151e400b4582b6"
Jul 29 15:10:18.352639 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:18.352625    2563 scope.go:117] "RemoveContainer" containerID="228c116ef10727eb1269278b06b165a07deef9f9c6241dc56e151e400b4582b6"
Jul 29 15:10:18.352901 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:10:18.352882    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"228c116ef10727eb1269278b06b165a07deef9f9c6241dc56e151e400b4582b6\": container with ID starting with 228c116ef10727eb1269278b06b165a07deef9f9c6241dc56e151e400b4582b6 not found: ID does not exist" containerID="228c116ef10727eb1269278b06b165a07deef9f9c6241dc56e151e400b4582b6"
Jul 29 15:10:18.352969 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:18.352906    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"228c116ef10727eb1269278b06b165a07deef9f9c6241dc56e151e400b4582b6"} err="failed to get container status \"228c116ef10727eb1269278b06b165a07deef9f9c6241dc56e151e400b4582b6\": rpc error: code = NotFound desc = could not find container \"228c116ef10727eb1269278b06b165a07deef9f9c6241dc56e151e400b4582b6\": container with ID starting with 228c116ef10727eb1269278b06b165a07deef9f9c6241dc56e151e400b4582b6 not found: ID does not exist"
Jul 29 15:10:18.364013 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:18.363991    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a385d8ab-f48b9d6cb-vcnzw"]
Jul 29 15:10:18.367880 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:18.367861    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a385d8ab-f48b9d6cb-vcnzw"]
Jul 29 15:10:18.432178 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:18.432101    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3d287fd-cc0b-47a9-a3b6-86aada844176" path="/var/lib/kubelet/pods/e3d287fd-cc0b-47a9-a3b6-86aada844176/volumes"
Jul 29 15:10:18.432458 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:18.432445    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdf3ea83-3a47-4c8a-ad54-7dde8d82a41e" path="/var/lib/kubelet/pods/fdf3ea83-3a47-4c8a-ad54-7dde8d82a41e/volumes"
Jul 29 15:10:58.831528 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:58.831400    2563 scope.go:117] "RemoveContainer" containerID="b39a25005a7e8eeecaa2906e9a76a6197d47bf93480a439b1c33ad5a50008ef1"
Jul 29 15:10:58.839508 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:10:58.839487    2563 scope.go:117] "RemoveContainer" containerID="3bf765168f0e3c4839649b638e487865ea1b6bc3d0daea22f426b51d5cb1c3e2"
Jul 29 15:11:15.765451 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:15.765411    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"]
Jul 29 15:11:15.765870 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:15.765629    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2" podUID="0847c0bd-0e85-447f-99c0-81552162797d" containerName="istio-proxy" containerID="cri-o://4a540a79a233b10e4033565f5ad60fea6f9a50dd42c882e4beb2197606f4806d" gracePeriod=30
Jul 29 15:11:21.020930 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:21.020905    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"
Jul 29 15:11:21.153070 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:21.152997    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0847c0bd-0e85-447f-99c0-81552162797d-istio-envoy\") pod \"0847c0bd-0e85-447f-99c0-81552162797d\" (UID: \"0847c0bd-0e85-447f-99c0-81552162797d\") "
Jul 29 15:11:21.153070 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:21.153034    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0847c0bd-0e85-447f-99c0-81552162797d-istio-data\") pod \"0847c0bd-0e85-447f-99c0-81552162797d\" (UID: \"0847c0bd-0e85-447f-99c0-81552162797d\") "
Jul 29 15:11:21.153308 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:21.153082    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0847c0bd-0e85-447f-99c0-81552162797d-workload-socket\") pod \"0847c0bd-0e85-447f-99c0-81552162797d\" (UID: \"0847c0bd-0e85-447f-99c0-81552162797d\") "
Jul 29 15:11:21.153308 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:21.153110    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0847c0bd-0e85-447f-99c0-81552162797d-istiod-ca-cert\") pod \"0847c0bd-0e85-447f-99c0-81552162797d\" (UID: \"0847c0bd-0e85-447f-99c0-81552162797d\") "
Jul 29 15:11:21.153308 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:21.153166    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0847c0bd-0e85-447f-99c0-81552162797d-credential-socket\") pod \"0847c0bd-0e85-447f-99c0-81552162797d\" (UID: \"0847c0bd-0e85-447f-99c0-81552162797d\") "
Jul 29 15:11:21.153308 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:21.153202    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0847c0bd-0e85-447f-99c0-81552162797d-workload-certs\") pod \"0847c0bd-0e85-447f-99c0-81552162797d\" (UID: \"0847c0bd-0e85-447f-99c0-81552162797d\") "
Jul 29 15:11:21.153308 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:21.153239    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5c65s\" (UniqueName: \"kubernetes.io/projected/0847c0bd-0e85-447f-99c0-81552162797d-kube-api-access-5c65s\") pod \"0847c0bd-0e85-447f-99c0-81552162797d\" (UID: \"0847c0bd-0e85-447f-99c0-81552162797d\") "
Jul 29 15:11:21.153308 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:21.153272    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0847c0bd-0e85-447f-99c0-81552162797d-istio-token\") pod \"0847c0bd-0e85-447f-99c0-81552162797d\" (UID: \"0847c0bd-0e85-447f-99c0-81552162797d\") "
Jul 29 15:11:21.153308 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:21.153296    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0847c0bd-0e85-447f-99c0-81552162797d-istio-podinfo\") pod \"0847c0bd-0e85-447f-99c0-81552162797d\" (UID: \"0847c0bd-0e85-447f-99c0-81552162797d\") "
Jul 29 15:11:21.153662 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:21.153447    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0847c0bd-0e85-447f-99c0-81552162797d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0847c0bd-0e85-447f-99c0-81552162797d" (UID: "0847c0bd-0e85-447f-99c0-81552162797d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:11:21.153662 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:21.153602    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0847c0bd-0e85-447f-99c0-81552162797d-workload-socket\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:11:21.153662 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:21.153619    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0847c0bd-0e85-447f-99c0-81552162797d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0847c0bd-0e85-447f-99c0-81552162797d" (UID: "0847c0bd-0e85-447f-99c0-81552162797d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:11:21.153662 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:21.153638    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0847c0bd-0e85-447f-99c0-81552162797d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0847c0bd-0e85-447f-99c0-81552162797d" (UID: "0847c0bd-0e85-447f-99c0-81552162797d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 15:11:21.153878 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:21.153687    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0847c0bd-0e85-447f-99c0-81552162797d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0847c0bd-0e85-447f-99c0-81552162797d" (UID: "0847c0bd-0e85-447f-99c0-81552162797d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:11:21.153878 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:21.153845    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0847c0bd-0e85-447f-99c0-81552162797d-istio-data" (OuterVolumeSpecName: "istio-data") pod "0847c0bd-0e85-447f-99c0-81552162797d" (UID: "0847c0bd-0e85-447f-99c0-81552162797d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:11:21.155472 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:21.155435    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0847c0bd-0e85-447f-99c0-81552162797d-kube-api-access-5c65s" (OuterVolumeSpecName: "kube-api-access-5c65s") pod "0847c0bd-0e85-447f-99c0-81552162797d" (UID: "0847c0bd-0e85-447f-99c0-81552162797d"). InnerVolumeSpecName "kube-api-access-5c65s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 15:11:21.155571 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:21.155553    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0847c0bd-0e85-447f-99c0-81552162797d-istio-token" (OuterVolumeSpecName: "istio-token") pod "0847c0bd-0e85-447f-99c0-81552162797d" (UID: "0847c0bd-0e85-447f-99c0-81552162797d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 15:11:21.155978 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:21.155956    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0847c0bd-0e85-447f-99c0-81552162797d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0847c0bd-0e85-447f-99c0-81552162797d" (UID: "0847c0bd-0e85-447f-99c0-81552162797d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:11:21.156068 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:21.155984    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0847c0bd-0e85-447f-99c0-81552162797d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0847c0bd-0e85-447f-99c0-81552162797d" (UID: "0847c0bd-0e85-447f-99c0-81552162797d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 15:11:21.254416 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:21.254390    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0847c0bd-0e85-447f-99c0-81552162797d-istiod-ca-cert\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:11:21.254416 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:21.254410    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0847c0bd-0e85-447f-99c0-81552162797d-credential-socket\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:11:21.254566 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:21.254422    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0847c0bd-0e85-447f-99c0-81552162797d-workload-certs\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:11:21.254566 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:21.254432    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c65s\" (UniqueName: \"kubernetes.io/projected/0847c0bd-0e85-447f-99c0-81552162797d-kube-api-access-5c65s\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:11:21.254566 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:21.254441    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0847c0bd-0e85-447f-99c0-81552162797d-istio-token\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:11:21.254566 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:21.254450    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0847c0bd-0e85-447f-99c0-81552162797d-istio-podinfo\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:11:21.254566 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:21.254457    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0847c0bd-0e85-447f-99c0-81552162797d-istio-envoy\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:11:21.254566 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:21.254464    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0847c0bd-0e85-447f-99c0-81552162797d-istio-data\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:11:21.666732 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:21.666695    2563 generic.go:356] "Generic (PLEG): container finished" podID="0847c0bd-0e85-447f-99c0-81552162797d" containerID="4a540a79a233b10e4033565f5ad60fea6f9a50dd42c882e4beb2197606f4806d" exitCode=0
Jul 29 15:11:21.666891 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:21.666751    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"
Jul 29 15:11:21.666891 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:21.666781    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2" event={"ID":"0847c0bd-0e85-447f-99c0-81552162797d","Type":"ContainerDied","Data":"4a540a79a233b10e4033565f5ad60fea6f9a50dd42c882e4beb2197606f4806d"}
Jul 29 15:11:21.666891 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:21.666858    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2" event={"ID":"0847c0bd-0e85-447f-99c0-81552162797d","Type":"ContainerDied","Data":"5f33bd0fba0ed8fa3b21eb7afba1cef1fc3ce4fde06aac539dd07dff58c5bfd1"}
Jul 29 15:11:21.666891 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:21.666883    2563 scope.go:117] "RemoveContainer" containerID="4a540a79a233b10e4033565f5ad60fea6f9a50dd42c882e4beb2197606f4806d"
Jul 29 15:11:21.676386 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:21.676368    2563 scope.go:117] "RemoveContainer" containerID="bc3d19f0c647f0c92a5ceffdaa5dd58f132d04ccdf7835b296e7115c0c176cb5"
Jul 29 15:11:21.683871 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:21.683855    2563 scope.go:117] "RemoveContainer" containerID="4a540a79a233b10e4033565f5ad60fea6f9a50dd42c882e4beb2197606f4806d"
Jul 29 15:11:21.684153 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:11:21.684114    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a540a79a233b10e4033565f5ad60fea6f9a50dd42c882e4beb2197606f4806d\": container with ID starting with 4a540a79a233b10e4033565f5ad60fea6f9a50dd42c882e4beb2197606f4806d not found: ID does not exist" containerID="4a540a79a233b10e4033565f5ad60fea6f9a50dd42c882e4beb2197606f4806d"
Jul 29 15:11:21.684230 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:21.684152    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a540a79a233b10e4033565f5ad60fea6f9a50dd42c882e4beb2197606f4806d"} err="failed to get container status \"4a540a79a233b10e4033565f5ad60fea6f9a50dd42c882e4beb2197606f4806d\": rpc error: code = NotFound desc = could not find container \"4a540a79a233b10e4033565f5ad60fea6f9a50dd42c882e4beb2197606f4806d\": container with ID starting with 4a540a79a233b10e4033565f5ad60fea6f9a50dd42c882e4beb2197606f4806d not found: ID does not exist"
Jul 29 15:11:21.684230 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:21.684169    2563 scope.go:117] "RemoveContainer" containerID="bc3d19f0c647f0c92a5ceffdaa5dd58f132d04ccdf7835b296e7115c0c176cb5"
Jul 29 15:11:21.684394 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:11:21.684375    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc3d19f0c647f0c92a5ceffdaa5dd58f132d04ccdf7835b296e7115c0c176cb5\": container with ID starting with bc3d19f0c647f0c92a5ceffdaa5dd58f132d04ccdf7835b296e7115c0c176cb5 not found: ID does not exist" containerID="bc3d19f0c647f0c92a5ceffdaa5dd58f132d04ccdf7835b296e7115c0c176cb5"
Jul 29 15:11:21.684437 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:21.684400    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc3d19f0c647f0c92a5ceffdaa5dd58f132d04ccdf7835b296e7115c0c176cb5"} err="failed to get container status \"bc3d19f0c647f0c92a5ceffdaa5dd58f132d04ccdf7835b296e7115c0c176cb5\": rpc error: code = NotFound desc = could not find container \"bc3d19f0c647f0c92a5ceffdaa5dd58f132d04ccdf7835b296e7115c0c176cb5\": container with ID starting with bc3d19f0c647f0c92a5ceffdaa5dd58f132d04ccdf7835b296e7115c0c176cb5 not found: ID does not exist"
Jul 29 15:11:21.687097 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:21.687075    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"]
Jul 29 15:11:21.691338 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:21.691317    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a385d8ab-openshift-default-ffcb895cb-g8ld2"]
Jul 29 15:11:22.432246 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:22.432210    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0847c0bd-0e85-447f-99c0-81552162797d" path="/var/lib/kubelet/pods/0847c0bd-0e85-447f-99c0-81552162797d/volumes"
Jul 29 15:11:58.864615 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:11:58.864577    2563 scope.go:117] "RemoveContainer" containerID="3813bdfd8e0c28ca837cb68453b48977e149ca0ce0a67b0a96042eb67f6ef78c"
Jul 29 15:12:11.966618 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:11.966585    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"]
Jul 29 15:12:11.967070 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:11.966964    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0847c0bd-0e85-447f-99c0-81552162797d" containerName="istio-proxy"
Jul 29 15:12:11.967070 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:11.966978    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0847c0bd-0e85-447f-99c0-81552162797d" containerName="istio-proxy"
Jul 29 15:12:11.967070 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:11.966993    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdf3ea83-3a47-4c8a-ad54-7dde8d82a41e" containerName="revoke-keys"
Jul 29 15:12:11.967070 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:11.966999    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdf3ea83-3a47-4c8a-ad54-7dde8d82a41e" containerName="revoke-keys"
Jul 29 15:12:11.967070 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:11.967008    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3d287fd-cc0b-47a9-a3b6-86aada844176" containerName="maas-api"
Jul 29 15:12:11.967070 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:11.967012    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3d287fd-cc0b-47a9-a3b6-86aada844176" containerName="maas-api"
Jul 29 15:12:11.967280 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:11.967083    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdf3ea83-3a47-4c8a-ad54-7dde8d82a41e" containerName="revoke-keys"
Jul 29 15:12:11.967280 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:11.967091    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0847c0bd-0e85-447f-99c0-81552162797d" containerName="istio-proxy"
Jul 29 15:12:11.967280 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:11.967098    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0847c0bd-0e85-447f-99c0-81552162797d" containerName="istio-proxy"
Jul 29 15:12:11.967280 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:11.967106    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3d287fd-cc0b-47a9-a3b6-86aada844176" containerName="maas-api"
Jul 29 15:12:11.970089 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:11.970073    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"
Jul 29 15:12:11.972894 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:11.972854    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 29 15:12:11.973042 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:11.972928    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-5ef0d2-openshift-default-dockercfg-7vbh9\""
Jul 29 15:12:11.983834 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:11.983807    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"]
Jul 29 15:12:12.091182 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:12.091126    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7f37c21-d58e-4d57-800f-83e87c09a6c2-istio-envoy\") pod \"e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt\" (UID: \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\") " pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"
Jul 29 15:12:12.091352 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:12.091196    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f37c21-d58e-4d57-800f-83e87c09a6c2-workload-socket\") pod \"e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt\" (UID: \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\") " pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"
Jul 29 15:12:12.091352 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:12.091216    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7f37c21-d58e-4d57-800f-83e87c09a6c2-istiod-ca-cert\") pod \"e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt\" (UID: \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\") " pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"
Jul 29 15:12:12.091352 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:12.091240    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f37c21-d58e-4d57-800f-83e87c09a6c2-credential-socket\") pod \"e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt\" (UID: \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\") " pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"
Jul 29 15:12:12.091352 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:12.091263    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz4mj\" (UniqueName: \"kubernetes.io/projected/a7f37c21-d58e-4d57-800f-83e87c09a6c2-kube-api-access-vz4mj\") pod \"e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt\" (UID: \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\") " pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"
Jul 29 15:12:12.091352 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:12.091293    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7f37c21-d58e-4d57-800f-83e87c09a6c2-istio-data\") pod \"e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt\" (UID: \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\") " pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"
Jul 29 15:12:12.091352 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:12.091311    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7f37c21-d58e-4d57-800f-83e87c09a6c2-workload-certs\") pod \"e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt\" (UID: \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\") " pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"
Jul 29 15:12:12.091352 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:12.091338    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7f37c21-d58e-4d57-800f-83e87c09a6c2-istio-token\") pod \"e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt\" (UID: \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\") " pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"
Jul 29 15:12:12.091652 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:12.091393    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7f37c21-d58e-4d57-800f-83e87c09a6c2-istio-podinfo\") pod \"e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt\" (UID: \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\") " pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"
Jul 29 15:12:12.192311 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:12.192275    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz4mj\" (UniqueName: \"kubernetes.io/projected/a7f37c21-d58e-4d57-800f-83e87c09a6c2-kube-api-access-vz4mj\") pod \"e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt\" (UID: \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\") " pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"
Jul 29 15:12:12.192463 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:12.192320    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7f37c21-d58e-4d57-800f-83e87c09a6c2-istio-data\") pod \"e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt\" (UID: \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\") " pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"
Jul 29 15:12:12.192533 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:12.192509    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7f37c21-d58e-4d57-800f-83e87c09a6c2-workload-certs\") pod \"e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt\" (UID: \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\") " pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"
Jul 29 15:12:12.192592 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:12.192562    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7f37c21-d58e-4d57-800f-83e87c09a6c2-istio-token\") pod \"e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt\" (UID: \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\") " pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"
Jul 29 15:12:12.192652 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:12.192628    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7f37c21-d58e-4d57-800f-83e87c09a6c2-istio-podinfo\") pod \"e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt\" (UID: \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\") " pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"
Jul 29 15:12:12.192707 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:12.192670    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7f37c21-d58e-4d57-800f-83e87c09a6c2-istio-envoy\") pod \"e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt\" (UID: \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\") " pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"
Jul 29 15:12:12.192768 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:12.192709    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f37c21-d58e-4d57-800f-83e87c09a6c2-workload-socket\") pod \"e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt\" (UID: \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\") " pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"
Jul 29 15:12:12.192768 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:12.192737    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7f37c21-d58e-4d57-800f-83e87c09a6c2-istiod-ca-cert\") pod \"e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt\" (UID: \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\") " pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"
Jul 29 15:12:12.192870 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:12.192779    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f37c21-d58e-4d57-800f-83e87c09a6c2-credential-socket\") pod \"e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt\" (UID: \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\") " pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"
Jul 29 15:12:12.192924 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:12.192869    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7f37c21-d58e-4d57-800f-83e87c09a6c2-workload-certs\") pod \"e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt\" (UID: \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\") " pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"
Jul 29 15:12:12.192924 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:12.192876    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7f37c21-d58e-4d57-800f-83e87c09a6c2-istio-data\") pod \"e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt\" (UID: \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\") " pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"
Jul 29 15:12:12.193171 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:12.193149    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f37c21-d58e-4d57-800f-83e87c09a6c2-credential-socket\") pod \"e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt\" (UID: \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\") " pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"
Jul 29 15:12:12.193249 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:12.193202    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f37c21-d58e-4d57-800f-83e87c09a6c2-workload-socket\") pod \"e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt\" (UID: \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\") " pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"
Jul 29 15:12:12.193446 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:12.193429    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7f37c21-d58e-4d57-800f-83e87c09a6c2-istiod-ca-cert\") pod \"e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt\" (UID: \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\") " pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"
Jul 29 15:12:12.194871 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:12.194852    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7f37c21-d58e-4d57-800f-83e87c09a6c2-istio-envoy\") pod \"e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt\" (UID: \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\") " pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"
Jul 29 15:12:12.195143 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:12.195115    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7f37c21-d58e-4d57-800f-83e87c09a6c2-istio-podinfo\") pod \"e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt\" (UID: \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\") " pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"
Jul 29 15:12:12.200362 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:12.200340    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7f37c21-d58e-4d57-800f-83e87c09a6c2-istio-token\") pod \"e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt\" (UID: \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\") " pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"
Jul 29 15:12:12.200446 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:12.200381    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz4mj\" (UniqueName: \"kubernetes.io/projected/a7f37c21-d58e-4d57-800f-83e87c09a6c2-kube-api-access-vz4mj\") pod \"e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt\" (UID: \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\") " pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"
Jul 29 15:12:12.284615 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:12.284554    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"
Jul 29 15:12:12.422530 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:12.422352    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"]
Jul 29 15:12:12.424745 ip-10-0-136-196 kubenswrapper[2563]: W0729 15:12:12.424717    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7f37c21_d58e_4d57_800f_83e87c09a6c2.slice/crio-eb0ba9c7b134464e0d96cea0b286995ed54055e8395e97abbbdf4cec0790cfcf WatchSource:0}: Error finding container eb0ba9c7b134464e0d96cea0b286995ed54055e8395e97abbbdf4cec0790cfcf: Status 404 returned error can't find the container with id eb0ba9c7b134464e0d96cea0b286995ed54055e8395e97abbbdf4cec0790cfcf
Jul 29 15:12:12.426875 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:12.426843    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:12:12.426964 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:12.426907    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:12:12.426964 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:12.426938    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:12:12.912312 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:12.912278    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt" event={"ID":"a7f37c21-d58e-4d57-800f-83e87c09a6c2","Type":"ContainerStarted","Data":"2b885650775a7a3a68b213eb402427ca9c8d4cad7d3aa3ab8eb1c6517a5dc95d"}
Jul 29 15:12:12.912496 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:12.912358    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt" event={"ID":"a7f37c21-d58e-4d57-800f-83e87c09a6c2","Type":"ContainerStarted","Data":"eb0ba9c7b134464e0d96cea0b286995ed54055e8395e97abbbdf4cec0790cfcf"}
Jul 29 15:12:12.935641 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:12.935587    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt" podStartSLOduration=1.935571296 podStartE2EDuration="1.935571296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:12:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:12:12.933569777 +0000 UTC m=+2234.982027916" watchObservedRunningTime="2026-07-29 15:12:12.935571296 +0000 UTC m=+2234.984029235"
Jul 29 15:12:13.285299 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:13.285270    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"
Jul 29 15:12:13.290410 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:13.290386    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"
Jul 29 15:12:13.917440 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:13.917403    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"
Jul 29 15:12:13.918244 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:13.918226    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"
Jul 29 15:12:15.025417 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:15.025359    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5ef0d2-544d7c599d-ml6vb"]
Jul 29 15:12:15.025802 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:15.025755    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0847c0bd-0e85-447f-99c0-81552162797d" containerName="istio-proxy"
Jul 29 15:12:15.025802 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:15.025766    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0847c0bd-0e85-447f-99c0-81552162797d" containerName="istio-proxy"
Jul 29 15:12:15.029116 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:15.029086    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ef0d2-544d7c599d-ml6vb"
Jul 29 15:12:15.031370 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:15.031346    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-5ef0d2-serving-cert\""
Jul 29 15:12:15.044037 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:15.044012    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5ef0d2-544d7c599d-ml6vb"]
Jul 29 15:12:15.118790 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:15.118754    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4p5j\" (UniqueName: \"kubernetes.io/projected/e3b2f800-b872-4be3-a6c6-58a2f3ca76c2-kube-api-access-v4p5j\") pod \"maas-api-e2e-inf-a-5ef0d2-544d7c599d-ml6vb\" (UID: \"e3b2f800-b872-4be3-a6c6-58a2f3ca76c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ef0d2-544d7c599d-ml6vb"
Jul 29 15:12:15.118790 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:15.118792    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3b2f800-b872-4be3-a6c6-58a2f3ca76c2-maas-api-tls\") pod \"maas-api-e2e-inf-a-5ef0d2-544d7c599d-ml6vb\" (UID: \"e3b2f800-b872-4be3-a6c6-58a2f3ca76c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ef0d2-544d7c599d-ml6vb"
Jul 29 15:12:15.119010 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:15.118837    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3b2f800-b872-4be3-a6c6-58a2f3ca76c2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5ef0d2-544d7c599d-ml6vb\" (UID: \"e3b2f800-b872-4be3-a6c6-58a2f3ca76c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ef0d2-544d7c599d-ml6vb"
Jul 29 15:12:15.219396 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:15.219362    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4p5j\" (UniqueName: \"kubernetes.io/projected/e3b2f800-b872-4be3-a6c6-58a2f3ca76c2-kube-api-access-v4p5j\") pod \"maas-api-e2e-inf-a-5ef0d2-544d7c599d-ml6vb\" (UID: \"e3b2f800-b872-4be3-a6c6-58a2f3ca76c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ef0d2-544d7c599d-ml6vb"
Jul 29 15:12:15.219396 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:15.219405    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3b2f800-b872-4be3-a6c6-58a2f3ca76c2-maas-api-tls\") pod \"maas-api-e2e-inf-a-5ef0d2-544d7c599d-ml6vb\" (UID: \"e3b2f800-b872-4be3-a6c6-58a2f3ca76c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ef0d2-544d7c599d-ml6vb"
Jul 29 15:12:15.219681 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:15.219467    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3b2f800-b872-4be3-a6c6-58a2f3ca76c2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5ef0d2-544d7c599d-ml6vb\" (UID: \"e3b2f800-b872-4be3-a6c6-58a2f3ca76c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ef0d2-544d7c599d-ml6vb"
Jul 29 15:12:15.219681 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:12:15.219576    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-5ef0d2-serving-cert: secret "maas-api-e2e-inf-a-5ef0d2-serving-cert" not found
Jul 29 15:12:15.219681 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:12:15.219647    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3b2f800-b872-4be3-a6c6-58a2f3ca76c2-maas-api-tls podName:e3b2f800-b872-4be3-a6c6-58a2f3ca76c2 nodeName:}" failed. No retries permitted until 2026-07-29 15:12:15.719628237 +0000 UTC m=+2237.768086155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e3b2f800-b872-4be3-a6c6-58a2f3ca76c2-maas-api-tls") pod "maas-api-e2e-inf-a-5ef0d2-544d7c599d-ml6vb" (UID: "e3b2f800-b872-4be3-a6c6-58a2f3ca76c2") : secret "maas-api-e2e-inf-a-5ef0d2-serving-cert" not found
Jul 29 15:12:15.220212 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:15.220190    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3b2f800-b872-4be3-a6c6-58a2f3ca76c2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5ef0d2-544d7c599d-ml6vb\" (UID: \"e3b2f800-b872-4be3-a6c6-58a2f3ca76c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ef0d2-544d7c599d-ml6vb"
Jul 29 15:12:15.233109 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:15.233075    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4p5j\" (UniqueName: \"kubernetes.io/projected/e3b2f800-b872-4be3-a6c6-58a2f3ca76c2-kube-api-access-v4p5j\") pod \"maas-api-e2e-inf-a-5ef0d2-544d7c599d-ml6vb\" (UID: \"e3b2f800-b872-4be3-a6c6-58a2f3ca76c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ef0d2-544d7c599d-ml6vb"
Jul 29 15:12:15.723407 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:15.723368    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3b2f800-b872-4be3-a6c6-58a2f3ca76c2-maas-api-tls\") pod \"maas-api-e2e-inf-a-5ef0d2-544d7c599d-ml6vb\" (UID: \"e3b2f800-b872-4be3-a6c6-58a2f3ca76c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ef0d2-544d7c599d-ml6vb"
Jul 29 15:12:15.726056 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:15.726026    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3b2f800-b872-4be3-a6c6-58a2f3ca76c2-maas-api-tls\") pod \"maas-api-e2e-inf-a-5ef0d2-544d7c599d-ml6vb\" (UID: \"e3b2f800-b872-4be3-a6c6-58a2f3ca76c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ef0d2-544d7c599d-ml6vb"
Jul 29 15:12:15.943446 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:15.943410    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ef0d2-544d7c599d-ml6vb"
Jul 29 15:12:16.093475 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:16.093447    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5ef0d2-544d7c599d-ml6vb"]
Jul 29 15:12:16.095770 ip-10-0-136-196 kubenswrapper[2563]: W0729 15:12:16.095741    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3b2f800_b872_4be3_a6c6_58a2f3ca76c2.slice/crio-8a18ee83db7fbc988077d5ed5d54eedaf1fce585d3a74d4d3add259483dce6f3 WatchSource:0}: Error finding container 8a18ee83db7fbc988077d5ed5d54eedaf1fce585d3a74d4d3add259483dce6f3: Status 404 returned error can't find the container with id 8a18ee83db7fbc988077d5ed5d54eedaf1fce585d3a74d4d3add259483dce6f3
Jul 29 15:12:16.931737 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:16.931707    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ef0d2-544d7c599d-ml6vb" event={"ID":"e3b2f800-b872-4be3-a6c6-58a2f3ca76c2","Type":"ContainerStarted","Data":"8a18ee83db7fbc988077d5ed5d54eedaf1fce585d3a74d4d3add259483dce6f3"}
Jul 29 15:12:17.938202 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:17.938162    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ef0d2-544d7c599d-ml6vb" event={"ID":"e3b2f800-b872-4be3-a6c6-58a2f3ca76c2","Type":"ContainerStarted","Data":"d4e52d8ad258b4d589875cedb99d2ac834a648c5fdece050d23dc317cad79925"}
Jul 29 15:12:17.938666 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:17.938314    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ef0d2-544d7c599d-ml6vb"
Jul 29 15:12:17.957050 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:17.956995    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ef0d2-544d7c599d-ml6vb" podStartSLOduration=2.053739955 podStartE2EDuration="2.95698187s" totalImagesPullingTime="903.241915ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:12:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:12:17.955607811 +0000 UTC m=+2240.004065749" watchObservedRunningTime="2026-07-29 15:12:17.95698187 +0000 UTC m=+2240.005439809"
Jul 29 15:12:23.959315 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:23.958389    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ef0d2-544d7c599d-ml6vb"
Jul 29 15:12:29.959345 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:29.959301    2563 patch_prober.go:28] interesting pod/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 29 15:12:29.959741 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:29.959386    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt" podUID="a7f37c21-d58e-4d57-800f-83e87c09a6c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 29 15:12:31.003866 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:31.003832    2563 generic.go:356] "Generic (PLEG): container finished" podID="a7f37c21-d58e-4d57-800f-83e87c09a6c2" containerID="2b885650775a7a3a68b213eb402427ca9c8d4cad7d3aa3ab8eb1c6517a5dc95d" exitCode=0
Jul 29 15:12:31.004315 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:31.003875    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt" event={"ID":"a7f37c21-d58e-4d57-800f-83e87c09a6c2","Type":"ContainerDied","Data":"2b885650775a7a3a68b213eb402427ca9c8d4cad7d3aa3ab8eb1c6517a5dc95d"}
Jul 29 15:12:31.004315 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:31.004284    2563 scope.go:117] "RemoveContainer" containerID="2b885650775a7a3a68b213eb402427ca9c8d4cad7d3aa3ab8eb1c6517a5dc95d"
Jul 29 15:12:31.005322 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:31.005284    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:12:31.005448 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:31.005351    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:12:31.005448 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:31.005381    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:12:32.011654 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:32.011563    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt" event={"ID":"a7f37c21-d58e-4d57-800f-83e87c09a6c2","Type":"ContainerStarted","Data":"f9f0ec9c9b7a4bef0f041f369e0f100813ca7d53adf1a6468015671b3b1a7a8c"}
Jul 29 15:12:32.285355 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:32.285260    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"
Jul 29 15:12:32.285564 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:32.285540    2563 patch_prober.go:28] interesting pod/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Jul 29 15:12:32.285650 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:32.285598    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt" podUID="a7f37c21-d58e-4d57-800f-83e87c09a6c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Jul 29 15:12:33.285214 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:33.285178    2563 patch_prober.go:28] interesting pod/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Jul 29 15:12:33.285606 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:33.285235    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt" podUID="a7f37c21-d58e-4d57-800f-83e87c09a6c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Jul 29 15:12:34.285782 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:34.285744    2563 patch_prober.go:28] interesting pod/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Jul 29 15:12:34.286216 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:34.285800    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt" podUID="a7f37c21-d58e-4d57-800f-83e87c09a6c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Jul 29 15:12:35.284783 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:35.284742    2563 patch_prober.go:28] interesting pod/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Jul 29 15:12:35.284975 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:35.284799    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt" podUID="a7f37c21-d58e-4d57-800f-83e87c09a6c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Jul 29 15:12:36.289246 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:36.289210    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"
Jul 29 15:12:36.289621 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:36.289589    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"
Jul 29 15:12:36.290264 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:36.290243    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"
Jul 29 15:12:40.754684 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:40.754641    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"]
Jul 29 15:12:40.757762 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:40.757740    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"
Jul 29 15:12:40.759678 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:40.759657    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-80fc9d-openshift-default-dockercfg-sflt2\""
Jul 29 15:12:40.773439 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:40.773415    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"]
Jul 29 15:12:40.849026 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:40.848990    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dbd8548-c376-46f9-925a-4f24ef7f021a-istio-token\") pod \"e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg\" (UID: \"5dbd8548-c376-46f9-925a-4f24ef7f021a\") " pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"
Jul 29 15:12:40.849241 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:40.849036    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dbd8548-c376-46f9-925a-4f24ef7f021a-workload-certs\") pod \"e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg\" (UID: \"5dbd8548-c376-46f9-925a-4f24ef7f021a\") " pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"
Jul 29 15:12:40.849241 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:40.849068    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbd8548-c376-46f9-925a-4f24ef7f021a-workload-socket\") pod \"e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg\" (UID: \"5dbd8548-c376-46f9-925a-4f24ef7f021a\") " pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"
Jul 29 15:12:40.849241 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:40.849163    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dbd8548-c376-46f9-925a-4f24ef7f021a-istio-envoy\") pod \"e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg\" (UID: \"5dbd8548-c376-46f9-925a-4f24ef7f021a\") " pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"
Jul 29 15:12:40.849241 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:40.849208    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49x5z\" (UniqueName: \"kubernetes.io/projected/5dbd8548-c376-46f9-925a-4f24ef7f021a-kube-api-access-49x5z\") pod \"e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg\" (UID: \"5dbd8548-c376-46f9-925a-4f24ef7f021a\") " pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"
Jul 29 15:12:40.849425 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:40.849280    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dbd8548-c376-46f9-925a-4f24ef7f021a-istiod-ca-cert\") pod \"e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg\" (UID: \"5dbd8548-c376-46f9-925a-4f24ef7f021a\") " pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"
Jul 29 15:12:40.849425 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:40.849324    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbd8548-c376-46f9-925a-4f24ef7f021a-credential-socket\") pod \"e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg\" (UID: \"5dbd8548-c376-46f9-925a-4f24ef7f021a\") " pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"
Jul 29 15:12:40.849425 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:40.849355    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dbd8548-c376-46f9-925a-4f24ef7f021a-istio-data\") pod \"e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg\" (UID: \"5dbd8548-c376-46f9-925a-4f24ef7f021a\") " pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"
Jul 29 15:12:40.849425 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:40.849382    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dbd8548-c376-46f9-925a-4f24ef7f021a-istio-podinfo\") pod \"e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg\" (UID: \"5dbd8548-c376-46f9-925a-4f24ef7f021a\") " pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"
Jul 29 15:12:40.950165 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:40.950089    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbd8548-c376-46f9-925a-4f24ef7f021a-workload-socket\") pod \"e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg\" (UID: \"5dbd8548-c376-46f9-925a-4f24ef7f021a\") " pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"
Jul 29 15:12:40.950165 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:40.950171    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dbd8548-c376-46f9-925a-4f24ef7f021a-istio-envoy\") pod \"e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg\" (UID: \"5dbd8548-c376-46f9-925a-4f24ef7f021a\") " pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"
Jul 29 15:12:40.950441 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:40.950199    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49x5z\" (UniqueName: \"kubernetes.io/projected/5dbd8548-c376-46f9-925a-4f24ef7f021a-kube-api-access-49x5z\") pod \"e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg\" (UID: \"5dbd8548-c376-46f9-925a-4f24ef7f021a\") " pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"
Jul 29 15:12:40.950441 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:40.950257    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dbd8548-c376-46f9-925a-4f24ef7f021a-istiod-ca-cert\") pod \"e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg\" (UID: \"5dbd8548-c376-46f9-925a-4f24ef7f021a\") " pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"
Jul 29 15:12:40.950441 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:40.950292    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbd8548-c376-46f9-925a-4f24ef7f021a-credential-socket\") pod \"e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg\" (UID: \"5dbd8548-c376-46f9-925a-4f24ef7f021a\") " pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"
Jul 29 15:12:40.950441 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:40.950319    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dbd8548-c376-46f9-925a-4f24ef7f021a-istio-data\") pod \"e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg\" (UID: \"5dbd8548-c376-46f9-925a-4f24ef7f021a\") " pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"
Jul 29 15:12:40.950441 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:40.950345    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dbd8548-c376-46f9-925a-4f24ef7f021a-istio-podinfo\") pod \"e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg\" (UID: \"5dbd8548-c376-46f9-925a-4f24ef7f021a\") " pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"
Jul 29 15:12:40.950862 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:40.950804    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbd8548-c376-46f9-925a-4f24ef7f021a-credential-socket\") pod \"e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg\" (UID: \"5dbd8548-c376-46f9-925a-4f24ef7f021a\") " pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"
Jul 29 15:12:40.950862 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:40.950709    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dbd8548-c376-46f9-925a-4f24ef7f021a-istio-data\") pod \"e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg\" (UID: \"5dbd8548-c376-46f9-925a-4f24ef7f021a\") " pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"
Jul 29 15:12:40.950862 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:40.950601    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbd8548-c376-46f9-925a-4f24ef7f021a-workload-socket\") pod \"e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg\" (UID: \"5dbd8548-c376-46f9-925a-4f24ef7f021a\") " pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"
Jul 29 15:12:40.951070 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:40.950816    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dbd8548-c376-46f9-925a-4f24ef7f021a-istio-token\") pod \"e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg\" (UID: \"5dbd8548-c376-46f9-925a-4f24ef7f021a\") " pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"
Jul 29 15:12:40.951070 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:40.950921    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dbd8548-c376-46f9-925a-4f24ef7f021a-workload-certs\") pod \"e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg\" (UID: \"5dbd8548-c376-46f9-925a-4f24ef7f021a\") " pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"
Jul 29 15:12:40.951070 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:40.951046    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dbd8548-c376-46f9-925a-4f24ef7f021a-istiod-ca-cert\") pod \"e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg\" (UID: \"5dbd8548-c376-46f9-925a-4f24ef7f021a\") " pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"
Jul 29 15:12:40.951225 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:40.951190    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dbd8548-c376-46f9-925a-4f24ef7f021a-workload-certs\") pod \"e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg\" (UID: \"5dbd8548-c376-46f9-925a-4f24ef7f021a\") " pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"
Jul 29 15:12:40.952991 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:40.952971    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dbd8548-c376-46f9-925a-4f24ef7f021a-istio-envoy\") pod \"e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg\" (UID: \"5dbd8548-c376-46f9-925a-4f24ef7f021a\") " pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"
Jul 29 15:12:40.953152 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:40.953118    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dbd8548-c376-46f9-925a-4f24ef7f021a-istio-podinfo\") pod \"e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg\" (UID: \"5dbd8548-c376-46f9-925a-4f24ef7f021a\") " pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"
Jul 29 15:12:40.957523 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:40.957497    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49x5z\" (UniqueName: \"kubernetes.io/projected/5dbd8548-c376-46f9-925a-4f24ef7f021a-kube-api-access-49x5z\") pod \"e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg\" (UID: \"5dbd8548-c376-46f9-925a-4f24ef7f021a\") " pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"
Jul 29 15:12:40.957751 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:40.957731    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dbd8548-c376-46f9-925a-4f24ef7f021a-istio-token\") pod \"e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg\" (UID: \"5dbd8548-c376-46f9-925a-4f24ef7f021a\") " pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"
Jul 29 15:12:41.067950 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:41.067856    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"
Jul 29 15:12:41.223791 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:41.215421    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"]
Jul 29 15:12:41.223791 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:41.222856    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:12:41.223791 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:41.222997    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:12:41.223791 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:41.223087    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:12:42.065044 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:42.065002    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg" event={"ID":"5dbd8548-c376-46f9-925a-4f24ef7f021a","Type":"ContainerStarted","Data":"6196ded0ff6446e848a022d18fa7a84fe46b09067d29472e66f595f2c779ba1c"}
Jul 29 15:12:42.065500 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:42.065087    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg" event={"ID":"5dbd8548-c376-46f9-925a-4f24ef7f021a","Type":"ContainerStarted","Data":"322df3b5e531b9dca01a2d2b9d9d6c413e9630710e738325f0614e56d01d5f34"}
Jul 29 15:12:42.068416 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:42.068393    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"
Jul 29 15:12:42.073343 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:42.073322    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"
Jul 29 15:12:42.108447 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:42.108390    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg" podStartSLOduration=2.108372772 podStartE2EDuration="2.108372772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:12:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:12:42.105610788 +0000 UTC m=+2264.154068728" watchObservedRunningTime="2026-07-29 15:12:42.108372772 +0000 UTC m=+2264.156830753"
Jul 29 15:12:43.070689 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:43.070655    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"
Jul 29 15:12:43.071904 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:43.071876    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"
Jul 29 15:12:44.309404 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:44.309368    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz"]
Jul 29 15:12:44.313263 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:44.313236    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz"
Jul 29 15:12:44.315417 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:44.315389    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-80fc9d-serving-cert\""
Jul 29 15:12:44.325477 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:44.324164    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz"]
Jul 29 15:12:44.385100 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:44.385067    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ba9efc7-9fab-4472-90e1-d9433fd7f4f2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz\" (UID: \"0ba9efc7-9fab-4472-90e1-d9433fd7f4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz"
Jul 29 15:12:44.385100 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:44.385112    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ba9efc7-9fab-4472-90e1-d9433fd7f4f2-maas-api-tls\") pod \"maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz\" (UID: \"0ba9efc7-9fab-4472-90e1-d9433fd7f4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz"
Jul 29 15:12:44.385354 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:44.385268    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-648wv\" (UniqueName: \"kubernetes.io/projected/0ba9efc7-9fab-4472-90e1-d9433fd7f4f2-kube-api-access-648wv\") pod \"maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz\" (UID: \"0ba9efc7-9fab-4472-90e1-d9433fd7f4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz"
Jul 29 15:12:44.486606 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:44.486564    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ba9efc7-9fab-4472-90e1-d9433fd7f4f2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz\" (UID: \"0ba9efc7-9fab-4472-90e1-d9433fd7f4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz"
Jul 29 15:12:44.486606 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:44.486605    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ba9efc7-9fab-4472-90e1-d9433fd7f4f2-maas-api-tls\") pod \"maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz\" (UID: \"0ba9efc7-9fab-4472-90e1-d9433fd7f4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz"
Jul 29 15:12:44.486837 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:44.486660    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-648wv\" (UniqueName: \"kubernetes.io/projected/0ba9efc7-9fab-4472-90e1-d9433fd7f4f2-kube-api-access-648wv\") pod \"maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz\" (UID: \"0ba9efc7-9fab-4472-90e1-d9433fd7f4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz"
Jul 29 15:12:44.486837 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:12:44.486797    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-80fc9d-serving-cert: secret "maas-api-e2e-inf-b-80fc9d-serving-cert" not found
Jul 29 15:12:44.486917 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:12:44.486860    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ba9efc7-9fab-4472-90e1-d9433fd7f4f2-maas-api-tls podName:0ba9efc7-9fab-4472-90e1-d9433fd7f4f2 nodeName:}" failed. No retries permitted until 2026-07-29 15:12:44.986843811 +0000 UTC m=+2267.035301727 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0ba9efc7-9fab-4472-90e1-d9433fd7f4f2-maas-api-tls") pod "maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz" (UID: "0ba9efc7-9fab-4472-90e1-d9433fd7f4f2") : secret "maas-api-e2e-inf-b-80fc9d-serving-cert" not found
Jul 29 15:12:44.487426 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:44.487404    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ba9efc7-9fab-4472-90e1-d9433fd7f4f2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz\" (UID: \"0ba9efc7-9fab-4472-90e1-d9433fd7f4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz"
Jul 29 15:12:44.495209 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:44.495179    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-648wv\" (UniqueName: \"kubernetes.io/projected/0ba9efc7-9fab-4472-90e1-d9433fd7f4f2-kube-api-access-648wv\") pod \"maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz\" (UID: \"0ba9efc7-9fab-4472-90e1-d9433fd7f4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz"
Jul 29 15:12:44.991920 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:44.991881    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ba9efc7-9fab-4472-90e1-d9433fd7f4f2-maas-api-tls\") pod \"maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz\" (UID: \"0ba9efc7-9fab-4472-90e1-d9433fd7f4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz"
Jul 29 15:12:44.992098 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:12:44.992053    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-80fc9d-serving-cert: secret "maas-api-e2e-inf-b-80fc9d-serving-cert" not found
Jul 29 15:12:44.992185 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:12:44.992171    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ba9efc7-9fab-4472-90e1-d9433fd7f4f2-maas-api-tls podName:0ba9efc7-9fab-4472-90e1-d9433fd7f4f2 nodeName:}" failed. No retries permitted until 2026-07-29 15:12:45.992150238 +0000 UTC m=+2268.040608169 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0ba9efc7-9fab-4472-90e1-d9433fd7f4f2-maas-api-tls") pod "maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz" (UID: "0ba9efc7-9fab-4472-90e1-d9433fd7f4f2") : secret "maas-api-e2e-inf-b-80fc9d-serving-cert" not found
Jul 29 15:12:46.003906 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:46.003856    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ba9efc7-9fab-4472-90e1-d9433fd7f4f2-maas-api-tls\") pod \"maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz\" (UID: \"0ba9efc7-9fab-4472-90e1-d9433fd7f4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz"
Jul 29 15:12:46.006469 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:46.006441    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ba9efc7-9fab-4472-90e1-d9433fd7f4f2-maas-api-tls\") pod \"maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz\" (UID: \"0ba9efc7-9fab-4472-90e1-d9433fd7f4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz"
Jul 29 15:12:46.127250 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:46.127201    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz"
Jul 29 15:12:46.268934 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:46.268903    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz"]
Jul 29 15:12:46.271380 ip-10-0-136-196 kubenswrapper[2563]: W0729 15:12:46.271348    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ba9efc7_9fab_4472_90e1_d9433fd7f4f2.slice/crio-652d61c6117400ff8d3f22eaa3961652c5c31b64f827e3c284ce62a59dce675b WatchSource:0}: Error finding container 652d61c6117400ff8d3f22eaa3961652c5c31b64f827e3c284ce62a59dce675b: Status 404 returned error can't find the container with id 652d61c6117400ff8d3f22eaa3961652c5c31b64f827e3c284ce62a59dce675b
Jul 29 15:12:47.093452 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:47.093413    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz" event={"ID":"0ba9efc7-9fab-4472-90e1-d9433fd7f4f2","Type":"ContainerStarted","Data":"3d62c92fa0368e7d173be9c365245b38c45eb7b55bfb1011aaf092c0d9b1f8e3"}
Jul 29 15:12:47.093902 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:47.093465    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz" event={"ID":"0ba9efc7-9fab-4472-90e1-d9433fd7f4f2","Type":"ContainerStarted","Data":"652d61c6117400ff8d3f22eaa3961652c5c31b64f827e3c284ce62a59dce675b"}
Jul 29 15:12:47.093902 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:47.093515    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz"
Jul 29 15:12:47.113077 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:47.113022    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz" podStartSLOduration=2.691020231 podStartE2EDuration="3.113004755s" totalImagesPullingTime="421.984524ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:12:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:12:47.111968463 +0000 UTC m=+2269.160426402" watchObservedRunningTime="2026-07-29 15:12:47.113004755 +0000 UTC m=+2269.161462693"
Jul 29 15:12:53.103784 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:12:53.103756    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz"
Jul 29 15:13:02.183542 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:13:02.183505    2563 generic.go:356] "Generic (PLEG): container finished" podID="5dbd8548-c376-46f9-925a-4f24ef7f021a" containerID="6196ded0ff6446e848a022d18fa7a84fe46b09067d29472e66f595f2c779ba1c" exitCode=0
Jul 29 15:13:02.183958 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:13:02.183578    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg" event={"ID":"5dbd8548-c376-46f9-925a-4f24ef7f021a","Type":"ContainerDied","Data":"6196ded0ff6446e848a022d18fa7a84fe46b09067d29472e66f595f2c779ba1c"}
Jul 29 15:13:02.184003 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:13:02.183968    2563 scope.go:117] "RemoveContainer" containerID="6196ded0ff6446e848a022d18fa7a84fe46b09067d29472e66f595f2c779ba1c"
Jul 29 15:13:02.185021 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:13:02.184993    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:13:02.185123 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:13:02.185074    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:13:02.185226 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:13:02.185125    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:13:03.190211 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:13:03.190177    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg" event={"ID":"5dbd8548-c376-46f9-925a-4f24ef7f021a","Type":"ContainerStarted","Data":"2a5f053ec2380e5d2d77e355179cf6507eeaebe78e0040da0d1e3f6828d4be09"}
Jul 29 15:13:04.068648 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:13:04.068612    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"
Jul 29 15:13:04.069048 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:13:04.069022    2563 patch_prober.go:28] interesting pod/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Jul 29 15:13:04.069532 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:13:04.069234    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg" podUID="5dbd8548-c376-46f9-925a-4f24ef7f021a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Jul 29 15:13:05.069403 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:13:05.069352    2563 patch_prober.go:28] interesting pod/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Jul 29 15:13:05.069913 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:13:05.069442    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg" podUID="5dbd8548-c376-46f9-925a-4f24ef7f021a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Jul 29 15:13:06.068307 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:13:06.068259    2563 patch_prober.go:28] interesting pod/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Jul 29 15:13:06.068515 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:13:06.068345    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg" podUID="5dbd8548-c376-46f9-925a-4f24ef7f021a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Jul 29 15:13:07.072020 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:13:07.071988    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"
Jul 29 15:13:07.072411 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:13:07.072372    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"
Jul 29 15:13:07.073053 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:13:07.073036    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"
Jul 29 15:13:58.895028 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:13:58.894954    2563 scope.go:117] "RemoveContainer" containerID="e556cd9e28d5bc669599cb78ef678cdf223ddd6fca09c244d8a3feea9e4cefb1"
Jul 29 15:14:51.103007 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:14:51.102970    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5ef0d2-544d7c599d-ml6vb"]
Jul 29 15:14:51.103546 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:14:51.103212    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ef0d2-544d7c599d-ml6vb" podUID="e3b2f800-b872-4be3-a6c6-58a2f3ca76c2" containerName="maas-api" containerID="cri-o://d4e52d8ad258b4d589875cedb99d2ac834a648c5fdece050d23dc317cad79925" gracePeriod=30
Jul 29 15:14:51.367183 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:14:51.367109    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ef0d2-544d7c599d-ml6vb"
Jul 29 15:14:51.415414 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:14:51.415379    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4p5j\" (UniqueName: \"kubernetes.io/projected/e3b2f800-b872-4be3-a6c6-58a2f3ca76c2-kube-api-access-v4p5j\") pod \"e3b2f800-b872-4be3-a6c6-58a2f3ca76c2\" (UID: \"e3b2f800-b872-4be3-a6c6-58a2f3ca76c2\") "
Jul 29 15:14:51.415586 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:14:51.415467    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3b2f800-b872-4be3-a6c6-58a2f3ca76c2-openshift-service-ca\") pod \"e3b2f800-b872-4be3-a6c6-58a2f3ca76c2\" (UID: \"e3b2f800-b872-4be3-a6c6-58a2f3ca76c2\") "
Jul 29 15:14:51.415586 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:14:51.415501    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3b2f800-b872-4be3-a6c6-58a2f3ca76c2-maas-api-tls\") pod \"e3b2f800-b872-4be3-a6c6-58a2f3ca76c2\" (UID: \"e3b2f800-b872-4be3-a6c6-58a2f3ca76c2\") "
Jul 29 15:14:51.415882 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:14:51.415854    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3b2f800-b872-4be3-a6c6-58a2f3ca76c2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e3b2f800-b872-4be3-a6c6-58a2f3ca76c2" (UID: "e3b2f800-b872-4be3-a6c6-58a2f3ca76c2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 15:14:51.417561 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:14:51.417536    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3b2f800-b872-4be3-a6c6-58a2f3ca76c2-kube-api-access-v4p5j" (OuterVolumeSpecName: "kube-api-access-v4p5j") pod "e3b2f800-b872-4be3-a6c6-58a2f3ca76c2" (UID: "e3b2f800-b872-4be3-a6c6-58a2f3ca76c2"). InnerVolumeSpecName "kube-api-access-v4p5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 15:14:51.417683 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:14:51.417664    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3b2f800-b872-4be3-a6c6-58a2f3ca76c2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e3b2f800-b872-4be3-a6c6-58a2f3ca76c2" (UID: "e3b2f800-b872-4be3-a6c6-58a2f3ca76c2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 15:14:51.516943 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:14:51.516914    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3b2f800-b872-4be3-a6c6-58a2f3ca76c2-openshift-service-ca\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:14:51.516943 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:14:51.516941    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3b2f800-b872-4be3-a6c6-58a2f3ca76c2-maas-api-tls\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:14:51.517095 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:14:51.516952    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4p5j\" (UniqueName: \"kubernetes.io/projected/e3b2f800-b872-4be3-a6c6-58a2f3ca76c2-kube-api-access-v4p5j\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:14:51.716035 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:14:51.716003    2563 generic.go:356] "Generic (PLEG): container finished" podID="e3b2f800-b872-4be3-a6c6-58a2f3ca76c2" containerID="d4e52d8ad258b4d589875cedb99d2ac834a648c5fdece050d23dc317cad79925" exitCode=0
Jul 29 15:14:51.716212 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:14:51.716055    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ef0d2-544d7c599d-ml6vb"
Jul 29 15:14:51.716212 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:14:51.716089    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ef0d2-544d7c599d-ml6vb" event={"ID":"e3b2f800-b872-4be3-a6c6-58a2f3ca76c2","Type":"ContainerDied","Data":"d4e52d8ad258b4d589875cedb99d2ac834a648c5fdece050d23dc317cad79925"}
Jul 29 15:14:51.716212 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:14:51.716161    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5ef0d2-544d7c599d-ml6vb" event={"ID":"e3b2f800-b872-4be3-a6c6-58a2f3ca76c2","Type":"ContainerDied","Data":"8a18ee83db7fbc988077d5ed5d54eedaf1fce585d3a74d4d3add259483dce6f3"}
Jul 29 15:14:51.716212 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:14:51.716185    2563 scope.go:117] "RemoveContainer" containerID="d4e52d8ad258b4d589875cedb99d2ac834a648c5fdece050d23dc317cad79925"
Jul 29 15:14:51.725283 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:14:51.725257    2563 scope.go:117] "RemoveContainer" containerID="d4e52d8ad258b4d589875cedb99d2ac834a648c5fdece050d23dc317cad79925"
Jul 29 15:14:51.725560 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:14:51.725533    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4e52d8ad258b4d589875cedb99d2ac834a648c5fdece050d23dc317cad79925\": container with ID starting with d4e52d8ad258b4d589875cedb99d2ac834a648c5fdece050d23dc317cad79925 not found: ID does not exist" containerID="d4e52d8ad258b4d589875cedb99d2ac834a648c5fdece050d23dc317cad79925"
Jul 29 15:14:51.725662 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:14:51.725566    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4e52d8ad258b4d589875cedb99d2ac834a648c5fdece050d23dc317cad79925"} err="failed to get container status \"d4e52d8ad258b4d589875cedb99d2ac834a648c5fdece050d23dc317cad79925\": rpc error: code = NotFound desc = could not find container \"d4e52d8ad258b4d589875cedb99d2ac834a648c5fdece050d23dc317cad79925\": container with ID starting with d4e52d8ad258b4d589875cedb99d2ac834a648c5fdece050d23dc317cad79925 not found: ID does not exist"
Jul 29 15:14:51.738608 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:14:51.738580    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5ef0d2-544d7c599d-ml6vb"]
Jul 29 15:14:51.745011 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:14:51.744990    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5ef0d2-544d7c599d-ml6vb"]
Jul 29 15:14:52.432218 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:14:52.432184    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3b2f800-b872-4be3-a6c6-58a2f3ca76c2" path="/var/lib/kubelet/pods/e3b2f800-b872-4be3-a6c6-58a2f3ca76c2/volumes"
Jul 29 15:14:57.945417 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:14:57.945378    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"]
Jul 29 15:14:57.945831 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:14:57.945626    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt" podUID="a7f37c21-d58e-4d57-800f-83e87c09a6c2" containerName="istio-proxy" containerID="cri-o://f9f0ec9c9b7a4bef0f041f369e0f100813ca7d53adf1a6468015671b3b1a7a8c" gracePeriod=30
Jul 29 15:14:58.844454 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:14:58.844421    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xnm95_23d79588-937d-4c6d-8760-a081024f608e/console-operator/1.log"
Jul 29 15:14:58.845008 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:14:58.844988    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xnm95_23d79588-937d-4c6d-8760-a081024f608e/console-operator/1.log"
Jul 29 15:15:03.195340 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:03.195310    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"
Jul 29 15:15:03.214676 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:03.214607    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f37c21-d58e-4d57-800f-83e87c09a6c2-workload-socket\") pod \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\" (UID: \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\") "
Jul 29 15:15:03.214676 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:03.214642    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7f37c21-d58e-4d57-800f-83e87c09a6c2-workload-certs\") pod \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\" (UID: \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\") "
Jul 29 15:15:03.214676 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:03.214662    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vz4mj\" (UniqueName: \"kubernetes.io/projected/a7f37c21-d58e-4d57-800f-83e87c09a6c2-kube-api-access-vz4mj\") pod \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\" (UID: \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\") "
Jul 29 15:15:03.214925 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:03.214684    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7f37c21-d58e-4d57-800f-83e87c09a6c2-istio-token\") pod \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\" (UID: \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\") "
Jul 29 15:15:03.214925 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:03.214712    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f37c21-d58e-4d57-800f-83e87c09a6c2-credential-socket\") pod \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\" (UID: \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\") "
Jul 29 15:15:03.214925 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:03.214753    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7f37c21-d58e-4d57-800f-83e87c09a6c2-istiod-ca-cert\") pod \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\" (UID: \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\") "
Jul 29 15:15:03.214925 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:03.214828    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7f37c21-d58e-4d57-800f-83e87c09a6c2-istio-envoy\") pod \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\" (UID: \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\") "
Jul 29 15:15:03.214925 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:03.214852    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7f37c21-d58e-4d57-800f-83e87c09a6c2-istio-podinfo\") pod \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\" (UID: \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\") "
Jul 29 15:15:03.214925 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:03.214921    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7f37c21-d58e-4d57-800f-83e87c09a6c2-istio-data\") pod \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\" (UID: \"a7f37c21-d58e-4d57-800f-83e87c09a6c2\") "
Jul 29 15:15:03.215246 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:03.214940    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f37c21-d58e-4d57-800f-83e87c09a6c2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a7f37c21-d58e-4d57-800f-83e87c09a6c2" (UID: "a7f37c21-d58e-4d57-800f-83e87c09a6c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:15:03.215246 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:03.215066    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f37c21-d58e-4d57-800f-83e87c09a6c2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a7f37c21-d58e-4d57-800f-83e87c09a6c2" (UID: "a7f37c21-d58e-4d57-800f-83e87c09a6c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:15:03.215373 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:03.215252    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f37c21-d58e-4d57-800f-83e87c09a6c2-workload-socket\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:15:03.215373 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:03.215272    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f37c21-d58e-4d57-800f-83e87c09a6c2-credential-socket\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:15:03.215373 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:03.215294    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f37c21-d58e-4d57-800f-83e87c09a6c2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a7f37c21-d58e-4d57-800f-83e87c09a6c2" (UID: "a7f37c21-d58e-4d57-800f-83e87c09a6c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:15:03.215373 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:03.215355    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7f37c21-d58e-4d57-800f-83e87c09a6c2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a7f37c21-d58e-4d57-800f-83e87c09a6c2" (UID: "a7f37c21-d58e-4d57-800f-83e87c09a6c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 15:15:03.216106 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:03.216079    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f37c21-d58e-4d57-800f-83e87c09a6c2-istio-data" (OuterVolumeSpecName: "istio-data") pod "a7f37c21-d58e-4d57-800f-83e87c09a6c2" (UID: "a7f37c21-d58e-4d57-800f-83e87c09a6c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:15:03.217062 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:03.217024    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f37c21-d58e-4d57-800f-83e87c09a6c2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a7f37c21-d58e-4d57-800f-83e87c09a6c2" (UID: "a7f37c21-d58e-4d57-800f-83e87c09a6c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:15:03.217177 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:03.217076    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7f37c21-d58e-4d57-800f-83e87c09a6c2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a7f37c21-d58e-4d57-800f-83e87c09a6c2" (UID: "a7f37c21-d58e-4d57-800f-83e87c09a6c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 15:15:03.217283 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:03.217263    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7f37c21-d58e-4d57-800f-83e87c09a6c2-istio-token" (OuterVolumeSpecName: "istio-token") pod "a7f37c21-d58e-4d57-800f-83e87c09a6c2" (UID: "a7f37c21-d58e-4d57-800f-83e87c09a6c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 15:15:03.217367 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:03.217354    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7f37c21-d58e-4d57-800f-83e87c09a6c2-kube-api-access-vz4mj" (OuterVolumeSpecName: "kube-api-access-vz4mj") pod "a7f37c21-d58e-4d57-800f-83e87c09a6c2" (UID: "a7f37c21-d58e-4d57-800f-83e87c09a6c2"). InnerVolumeSpecName "kube-api-access-vz4mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 15:15:03.315933 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:03.315883    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7f37c21-d58e-4d57-800f-83e87c09a6c2-workload-certs\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:15:03.315933 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:03.315925    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz4mj\" (UniqueName: \"kubernetes.io/projected/a7f37c21-d58e-4d57-800f-83e87c09a6c2-kube-api-access-vz4mj\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:15:03.315933 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:03.315942    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7f37c21-d58e-4d57-800f-83e87c09a6c2-istio-token\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:15:03.316208 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:03.315954    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7f37c21-d58e-4d57-800f-83e87c09a6c2-istiod-ca-cert\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:15:03.316208 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:03.315966    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7f37c21-d58e-4d57-800f-83e87c09a6c2-istio-envoy\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:15:03.316208 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:03.315977    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7f37c21-d58e-4d57-800f-83e87c09a6c2-istio-podinfo\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:15:03.316208 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:03.315989    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7f37c21-d58e-4d57-800f-83e87c09a6c2-istio-data\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:15:03.775199 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:03.775166    2563 generic.go:356] "Generic (PLEG): container finished" podID="a7f37c21-d58e-4d57-800f-83e87c09a6c2" containerID="f9f0ec9c9b7a4bef0f041f369e0f100813ca7d53adf1a6468015671b3b1a7a8c" exitCode=0
Jul 29 15:15:03.775347 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:03.775260    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt" event={"ID":"a7f37c21-d58e-4d57-800f-83e87c09a6c2","Type":"ContainerDied","Data":"f9f0ec9c9b7a4bef0f041f369e0f100813ca7d53adf1a6468015671b3b1a7a8c"}
Jul 29 15:15:03.775347 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:03.775270    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"
Jul 29 15:15:03.775347 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:03.775314    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt" event={"ID":"a7f37c21-d58e-4d57-800f-83e87c09a6c2","Type":"ContainerDied","Data":"eb0ba9c7b134464e0d96cea0b286995ed54055e8395e97abbbdf4cec0790cfcf"}
Jul 29 15:15:03.775347 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:03.775344    2563 scope.go:117] "RemoveContainer" containerID="f9f0ec9c9b7a4bef0f041f369e0f100813ca7d53adf1a6468015671b3b1a7a8c"
Jul 29 15:15:03.784303 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:03.784284    2563 scope.go:117] "RemoveContainer" containerID="2b885650775a7a3a68b213eb402427ca9c8d4cad7d3aa3ab8eb1c6517a5dc95d"
Jul 29 15:15:03.794263 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:03.794229    2563 scope.go:117] "RemoveContainer" containerID="f9f0ec9c9b7a4bef0f041f369e0f100813ca7d53adf1a6468015671b3b1a7a8c"
Jul 29 15:15:03.794501 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:15:03.794484    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9f0ec9c9b7a4bef0f041f369e0f100813ca7d53adf1a6468015671b3b1a7a8c\": container with ID starting with f9f0ec9c9b7a4bef0f041f369e0f100813ca7d53adf1a6468015671b3b1a7a8c not found: ID does not exist" containerID="f9f0ec9c9b7a4bef0f041f369e0f100813ca7d53adf1a6468015671b3b1a7a8c"
Jul 29 15:15:03.794540 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:03.794509    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9f0ec9c9b7a4bef0f041f369e0f100813ca7d53adf1a6468015671b3b1a7a8c"} err="failed to get container status \"f9f0ec9c9b7a4bef0f041f369e0f100813ca7d53adf1a6468015671b3b1a7a8c\": rpc error: code = NotFound desc = could not find container \"f9f0ec9c9b7a4bef0f041f369e0f100813ca7d53adf1a6468015671b3b1a7a8c\": container with ID starting with f9f0ec9c9b7a4bef0f041f369e0f100813ca7d53adf1a6468015671b3b1a7a8c not found: ID does not exist"
Jul 29 15:15:03.794540 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:03.794524    2563 scope.go:117] "RemoveContainer" containerID="2b885650775a7a3a68b213eb402427ca9c8d4cad7d3aa3ab8eb1c6517a5dc95d"
Jul 29 15:15:03.794729 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:15:03.794714    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b885650775a7a3a68b213eb402427ca9c8d4cad7d3aa3ab8eb1c6517a5dc95d\": container with ID starting with 2b885650775a7a3a68b213eb402427ca9c8d4cad7d3aa3ab8eb1c6517a5dc95d not found: ID does not exist" containerID="2b885650775a7a3a68b213eb402427ca9c8d4cad7d3aa3ab8eb1c6517a5dc95d"
Jul 29 15:15:03.794764 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:03.794734    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b885650775a7a3a68b213eb402427ca9c8d4cad7d3aa3ab8eb1c6517a5dc95d"} err="failed to get container status \"2b885650775a7a3a68b213eb402427ca9c8d4cad7d3aa3ab8eb1c6517a5dc95d\": rpc error: code = NotFound desc = could not find container \"2b885650775a7a3a68b213eb402427ca9c8d4cad7d3aa3ab8eb1c6517a5dc95d\": container with ID starting with 2b885650775a7a3a68b213eb402427ca9c8d4cad7d3aa3ab8eb1c6517a5dc95d not found: ID does not exist"
Jul 29 15:15:03.798347 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:03.798326    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"]
Jul 29 15:15:03.802337 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:03.802312    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5ef0d2-openshift-default-fb57455d-sg9lt"]
Jul 29 15:15:04.432172 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:04.432120    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7f37c21-d58e-4d57-800f-83e87c09a6c2" path="/var/lib/kubelet/pods/a7f37c21-d58e-4d57-800f-83e87c09a6c2/volumes"
Jul 29 15:15:08.788105 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:08.788075    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz"]
Jul 29 15:15:08.788553 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:08.788304    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz" podUID="0ba9efc7-9fab-4472-90e1-d9433fd7f4f2" containerName="maas-api" containerID="cri-o://3d62c92fa0368e7d173be9c365245b38c45eb7b55bfb1011aaf092c0d9b1f8e3" gracePeriod=30
Jul 29 15:15:09.047893 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:09.047832    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz"
Jul 29 15:15:09.170671 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:09.170639    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ba9efc7-9fab-4472-90e1-d9433fd7f4f2-openshift-service-ca\") pod \"0ba9efc7-9fab-4472-90e1-d9433fd7f4f2\" (UID: \"0ba9efc7-9fab-4472-90e1-d9433fd7f4f2\") "
Jul 29 15:15:09.170827 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:09.170751    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-648wv\" (UniqueName: \"kubernetes.io/projected/0ba9efc7-9fab-4472-90e1-d9433fd7f4f2-kube-api-access-648wv\") pod \"0ba9efc7-9fab-4472-90e1-d9433fd7f4f2\" (UID: \"0ba9efc7-9fab-4472-90e1-d9433fd7f4f2\") "
Jul 29 15:15:09.170827 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:09.170785    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ba9efc7-9fab-4472-90e1-d9433fd7f4f2-maas-api-tls\") pod \"0ba9efc7-9fab-4472-90e1-d9433fd7f4f2\" (UID: \"0ba9efc7-9fab-4472-90e1-d9433fd7f4f2\") "
Jul 29 15:15:09.171063 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:09.171036    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ba9efc7-9fab-4472-90e1-d9433fd7f4f2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0ba9efc7-9fab-4472-90e1-d9433fd7f4f2" (UID: "0ba9efc7-9fab-4472-90e1-d9433fd7f4f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 15:15:09.172912 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:09.172881    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ba9efc7-9fab-4472-90e1-d9433fd7f4f2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0ba9efc7-9fab-4472-90e1-d9433fd7f4f2" (UID: "0ba9efc7-9fab-4472-90e1-d9433fd7f4f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 15:15:09.173013 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:09.172956    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ba9efc7-9fab-4472-90e1-d9433fd7f4f2-kube-api-access-648wv" (OuterVolumeSpecName: "kube-api-access-648wv") pod "0ba9efc7-9fab-4472-90e1-d9433fd7f4f2" (UID: "0ba9efc7-9fab-4472-90e1-d9433fd7f4f2"). InnerVolumeSpecName "kube-api-access-648wv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 15:15:09.272367 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:09.272324    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ba9efc7-9fab-4472-90e1-d9433fd7f4f2-openshift-service-ca\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:15:09.272367 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:09.272361    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-648wv\" (UniqueName: \"kubernetes.io/projected/0ba9efc7-9fab-4472-90e1-d9433fd7f4f2-kube-api-access-648wv\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:15:09.272367 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:09.272372    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ba9efc7-9fab-4472-90e1-d9433fd7f4f2-maas-api-tls\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:15:09.807092 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:09.807047    2563 generic.go:356] "Generic (PLEG): container finished" podID="0ba9efc7-9fab-4472-90e1-d9433fd7f4f2" containerID="3d62c92fa0368e7d173be9c365245b38c45eb7b55bfb1011aaf092c0d9b1f8e3" exitCode=0
Jul 29 15:15:09.807574 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:09.807097    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz"
Jul 29 15:15:09.807574 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:09.807111    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz" event={"ID":"0ba9efc7-9fab-4472-90e1-d9433fd7f4f2","Type":"ContainerDied","Data":"3d62c92fa0368e7d173be9c365245b38c45eb7b55bfb1011aaf092c0d9b1f8e3"}
Jul 29 15:15:09.807574 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:09.807187    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz" event={"ID":"0ba9efc7-9fab-4472-90e1-d9433fd7f4f2","Type":"ContainerDied","Data":"652d61c6117400ff8d3f22eaa3961652c5c31b64f827e3c284ce62a59dce675b"}
Jul 29 15:15:09.807574 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:09.807223    2563 scope.go:117] "RemoveContainer" containerID="3d62c92fa0368e7d173be9c365245b38c45eb7b55bfb1011aaf092c0d9b1f8e3"
Jul 29 15:15:09.816636 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:09.816618    2563 scope.go:117] "RemoveContainer" containerID="3d62c92fa0368e7d173be9c365245b38c45eb7b55bfb1011aaf092c0d9b1f8e3"
Jul 29 15:15:09.816894 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:15:09.816877    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d62c92fa0368e7d173be9c365245b38c45eb7b55bfb1011aaf092c0d9b1f8e3\": container with ID starting with 3d62c92fa0368e7d173be9c365245b38c45eb7b55bfb1011aaf092c0d9b1f8e3 not found: ID does not exist" containerID="3d62c92fa0368e7d173be9c365245b38c45eb7b55bfb1011aaf092c0d9b1f8e3"
Jul 29 15:15:09.816939 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:09.816902    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d62c92fa0368e7d173be9c365245b38c45eb7b55bfb1011aaf092c0d9b1f8e3"} err="failed to get container status \"3d62c92fa0368e7d173be9c365245b38c45eb7b55bfb1011aaf092c0d9b1f8e3\": rpc error: code = NotFound desc = could not find container \"3d62c92fa0368e7d173be9c365245b38c45eb7b55bfb1011aaf092c0d9b1f8e3\": container with ID starting with 3d62c92fa0368e7d173be9c365245b38c45eb7b55bfb1011aaf092c0d9b1f8e3 not found: ID does not exist"
Jul 29 15:15:09.829592 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:09.829567    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz"]
Jul 29 15:15:09.833673 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:09.833652    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-80fc9d-5597c645b9-bmsjz"]
Jul 29 15:15:10.431954 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:10.431921    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ba9efc7-9fab-4472-90e1-d9433fd7f4f2" path="/var/lib/kubelet/pods/0ba9efc7-9fab-4472-90e1-d9433fd7f4f2/volumes"
Jul 29 15:15:15.496972 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:15.496936    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"]
Jul 29 15:15:15.497438 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:15.497202    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg" podUID="5dbd8548-c376-46f9-925a-4f24ef7f021a" containerName="istio-proxy" containerID="cri-o://2a5f053ec2380e5d2d77e355179cf6507eeaebe78e0040da0d1e3f6828d4be09" gracePeriod=30
Jul 29 15:15:16.384218 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.384182    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"]
Jul 29 15:15:16.384615 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.384599    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7f37c21-d58e-4d57-800f-83e87c09a6c2" containerName="istio-proxy"
Jul 29 15:15:16.384665 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.384618    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7f37c21-d58e-4d57-800f-83e87c09a6c2" containerName="istio-proxy"
Jul 29 15:15:16.384665 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.384632    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3b2f800-b872-4be3-a6c6-58a2f3ca76c2" containerName="maas-api"
Jul 29 15:15:16.384665 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.384639    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3b2f800-b872-4be3-a6c6-58a2f3ca76c2" containerName="maas-api"
Jul 29 15:15:16.384665 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.384649    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7f37c21-d58e-4d57-800f-83e87c09a6c2" containerName="istio-proxy"
Jul 29 15:15:16.384665 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.384656    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7f37c21-d58e-4d57-800f-83e87c09a6c2" containerName="istio-proxy"
Jul 29 15:15:16.384834 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.384670    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ba9efc7-9fab-4472-90e1-d9433fd7f4f2" containerName="maas-api"
Jul 29 15:15:16.384834 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.384675    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ba9efc7-9fab-4472-90e1-d9433fd7f4f2" containerName="maas-api"
Jul 29 15:15:16.384834 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.384745    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7f37c21-d58e-4d57-800f-83e87c09a6c2" containerName="istio-proxy"
Jul 29 15:15:16.384834 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.384753    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ba9efc7-9fab-4472-90e1-d9433fd7f4f2" containerName="maas-api"
Jul 29 15:15:16.384834 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.384763    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3b2f800-b872-4be3-a6c6-58a2f3ca76c2" containerName="maas-api"
Jul 29 15:15:16.389161 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.389127    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"
Jul 29 15:15:16.391227 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.391195    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-baf2cc-openshift-default-dockercfg-9fwdf\""
Jul 29 15:15:16.397600 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.397572    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"]
Jul 29 15:15:16.542491 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.542446    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d84a00f9-6797-48be-9870-75a860426818-istiod-ca-cert\") pod \"e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn\" (UID: \"d84a00f9-6797-48be-9870-75a860426818\") " pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"
Jul 29 15:15:16.542491 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.542491    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plkqc\" (UniqueName: \"kubernetes.io/projected/d84a00f9-6797-48be-9870-75a860426818-kube-api-access-plkqc\") pod \"e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn\" (UID: \"d84a00f9-6797-48be-9870-75a860426818\") " pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"
Jul 29 15:15:16.543001 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.542521    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84a00f9-6797-48be-9870-75a860426818-credential-socket\") pod \"e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn\" (UID: \"d84a00f9-6797-48be-9870-75a860426818\") " pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"
Jul 29 15:15:16.543001 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.542547    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d84a00f9-6797-48be-9870-75a860426818-istio-data\") pod \"e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn\" (UID: \"d84a00f9-6797-48be-9870-75a860426818\") " pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"
Jul 29 15:15:16.543001 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.542595    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d84a00f9-6797-48be-9870-75a860426818-istio-token\") pod \"e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn\" (UID: \"d84a00f9-6797-48be-9870-75a860426818\") " pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"
Jul 29 15:15:16.543001 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.542722    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d84a00f9-6797-48be-9870-75a860426818-workload-certs\") pod \"e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn\" (UID: \"d84a00f9-6797-48be-9870-75a860426818\") " pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"
Jul 29 15:15:16.543001 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.542761    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d84a00f9-6797-48be-9870-75a860426818-istio-podinfo\") pod \"e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn\" (UID: \"d84a00f9-6797-48be-9870-75a860426818\") " pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"
Jul 29 15:15:16.543001 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.542787    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84a00f9-6797-48be-9870-75a860426818-workload-socket\") pod \"e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn\" (UID: \"d84a00f9-6797-48be-9870-75a860426818\") " pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"
Jul 29 15:15:16.543001 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.542816    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d84a00f9-6797-48be-9870-75a860426818-istio-envoy\") pod \"e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn\" (UID: \"d84a00f9-6797-48be-9870-75a860426818\") " pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"
Jul 29 15:15:16.643995 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.643899    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d84a00f9-6797-48be-9870-75a860426818-istiod-ca-cert\") pod \"e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn\" (UID: \"d84a00f9-6797-48be-9870-75a860426818\") " pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"
Jul 29 15:15:16.643995 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.643937    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plkqc\" (UniqueName: \"kubernetes.io/projected/d84a00f9-6797-48be-9870-75a860426818-kube-api-access-plkqc\") pod \"e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn\" (UID: \"d84a00f9-6797-48be-9870-75a860426818\") " pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"
Jul 29 15:15:16.643995 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.643966    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84a00f9-6797-48be-9870-75a860426818-credential-socket\") pod \"e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn\" (UID: \"d84a00f9-6797-48be-9870-75a860426818\") " pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"
Jul 29 15:15:16.644326 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.644000    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d84a00f9-6797-48be-9870-75a860426818-istio-data\") pod \"e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn\" (UID: \"d84a00f9-6797-48be-9870-75a860426818\") " pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"
Jul 29 15:15:16.644326 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.644016    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d84a00f9-6797-48be-9870-75a860426818-istio-token\") pod \"e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn\" (UID: \"d84a00f9-6797-48be-9870-75a860426818\") " pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"
Jul 29 15:15:16.644326 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.644082    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d84a00f9-6797-48be-9870-75a860426818-workload-certs\") pod \"e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn\" (UID: \"d84a00f9-6797-48be-9870-75a860426818\") " pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"
Jul 29 15:15:16.644326 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.644119    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d84a00f9-6797-48be-9870-75a860426818-istio-podinfo\") pod \"e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn\" (UID: \"d84a00f9-6797-48be-9870-75a860426818\") " pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"
Jul 29 15:15:16.644326 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.644168    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84a00f9-6797-48be-9870-75a860426818-workload-socket\") pod \"e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn\" (UID: \"d84a00f9-6797-48be-9870-75a860426818\") " pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"
Jul 29 15:15:16.644326 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.644198    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d84a00f9-6797-48be-9870-75a860426818-istio-envoy\") pod \"e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn\" (UID: \"d84a00f9-6797-48be-9870-75a860426818\") " pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"
Jul 29 15:15:16.644619 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.644580    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d84a00f9-6797-48be-9870-75a860426818-istio-data\") pod \"e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn\" (UID: \"d84a00f9-6797-48be-9870-75a860426818\") " pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"
Jul 29 15:15:16.644670 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.644624    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84a00f9-6797-48be-9870-75a860426818-credential-socket\") pod \"e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn\" (UID: \"d84a00f9-6797-48be-9870-75a860426818\") " pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"
Jul 29 15:15:16.644807 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.644782    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d84a00f9-6797-48be-9870-75a860426818-istiod-ca-cert\") pod \"e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn\" (UID: \"d84a00f9-6797-48be-9870-75a860426818\") " pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"
Jul 29 15:15:16.644887 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.644865    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84a00f9-6797-48be-9870-75a860426818-workload-socket\") pod \"e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn\" (UID: \"d84a00f9-6797-48be-9870-75a860426818\") " pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"
Jul 29 15:15:16.644946 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.644907    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d84a00f9-6797-48be-9870-75a860426818-workload-certs\") pod \"e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn\" (UID: \"d84a00f9-6797-48be-9870-75a860426818\") " pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"
Jul 29 15:15:16.646578 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.646551    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d84a00f9-6797-48be-9870-75a860426818-istio-envoy\") pod \"e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn\" (UID: \"d84a00f9-6797-48be-9870-75a860426818\") " pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"
Jul 29 15:15:16.646818 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.646801    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d84a00f9-6797-48be-9870-75a860426818-istio-podinfo\") pod \"e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn\" (UID: \"d84a00f9-6797-48be-9870-75a860426818\") " pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"
Jul 29 15:15:16.651612 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.651593    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d84a00f9-6797-48be-9870-75a860426818-istio-token\") pod \"e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn\" (UID: \"d84a00f9-6797-48be-9870-75a860426818\") " pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"
Jul 29 15:15:16.651829 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.651813    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plkqc\" (UniqueName: \"kubernetes.io/projected/d84a00f9-6797-48be-9870-75a860426818-kube-api-access-plkqc\") pod \"e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn\" (UID: \"d84a00f9-6797-48be-9870-75a860426818\") " pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"
Jul 29 15:15:16.701570 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.701545    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"
Jul 29 15:15:16.844110 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.844031    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"]
Jul 29 15:15:16.846625 ip-10-0-136-196 kubenswrapper[2563]: W0729 15:15:16.846596    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd84a00f9_6797_48be_9870_75a860426818.slice/crio-d45bdcd7083b2d9356cea5559796ac0b09de7d60b26ec62c7861e7c2298e125b WatchSource:0}: Error finding container d45bdcd7083b2d9356cea5559796ac0b09de7d60b26ec62c7861e7c2298e125b: Status 404 returned error can't find the container with id d45bdcd7083b2d9356cea5559796ac0b09de7d60b26ec62c7861e7c2298e125b
Jul 29 15:15:16.848602 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.848570    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:15:16.848677 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.848634    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:15:16.848677 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:16.848662    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:15:17.846106 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:17.846072    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn" event={"ID":"d84a00f9-6797-48be-9870-75a860426818","Type":"ContainerStarted","Data":"c278cab064ee76ddc8e041d155f3d7a05ba6c5fa124ae8512441742f006083e6"}
Jul 29 15:15:17.846570 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:17.846156    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn" event={"ID":"d84a00f9-6797-48be-9870-75a860426818","Type":"ContainerStarted","Data":"d45bdcd7083b2d9356cea5559796ac0b09de7d60b26ec62c7861e7c2298e125b"}
Jul 29 15:15:17.874269 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:17.870324    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn" podStartSLOduration=1.8703047069999998 podStartE2EDuration="1.870304707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:15:17.866965871 +0000 UTC m=+2419.915423814" watchObservedRunningTime="2026-07-29 15:15:17.870304707 +0000 UTC m=+2419.918762648"
Jul 29 15:15:18.701882 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:18.701847    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"
Jul 29 15:15:18.706928 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:18.706903    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"
Jul 29 15:15:18.851167 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:18.851112    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"
Jul 29 15:15:18.852093 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:18.852076    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"
Jul 29 15:15:20.021800 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.021761    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-baf2cc-c47dfc746-csd9v"]
Jul 29 15:15:20.022439 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.022418    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7f37c21-d58e-4d57-800f-83e87c09a6c2" containerName="istio-proxy"
Jul 29 15:15:20.025886 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.025866    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-baf2cc-c47dfc746-csd9v"
Jul 29 15:15:20.029343 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.029318    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-baf2cc-serving-cert\""
Jul 29 15:15:20.036996 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.036974    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-baf2cc-c47dfc746-csd9v"]
Jul 29 15:15:20.177889 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.177851    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b000ba3c-f29c-4314-877d-15246ef14307-openshift-service-ca\") pod \"maas-api-e2e-api-a-baf2cc-c47dfc746-csd9v\" (UID: \"b000ba3c-f29c-4314-877d-15246ef14307\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-baf2cc-c47dfc746-csd9v"
Jul 29 15:15:20.178102 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.177909    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b000ba3c-f29c-4314-877d-15246ef14307-maas-api-tls\") pod \"maas-api-e2e-api-a-baf2cc-c47dfc746-csd9v\" (UID: \"b000ba3c-f29c-4314-877d-15246ef14307\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-baf2cc-c47dfc746-csd9v"
Jul 29 15:15:20.178102 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.177931    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxx72\" (UniqueName: \"kubernetes.io/projected/b000ba3c-f29c-4314-877d-15246ef14307-kube-api-access-nxx72\") pod \"maas-api-e2e-api-a-baf2cc-c47dfc746-csd9v\" (UID: \"b000ba3c-f29c-4314-877d-15246ef14307\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-baf2cc-c47dfc746-csd9v"
Jul 29 15:15:20.279014 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.278923    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b000ba3c-f29c-4314-877d-15246ef14307-openshift-service-ca\") pod \"maas-api-e2e-api-a-baf2cc-c47dfc746-csd9v\" (UID: \"b000ba3c-f29c-4314-877d-15246ef14307\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-baf2cc-c47dfc746-csd9v"
Jul 29 15:15:20.279014 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.278997    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b000ba3c-f29c-4314-877d-15246ef14307-maas-api-tls\") pod \"maas-api-e2e-api-a-baf2cc-c47dfc746-csd9v\" (UID: \"b000ba3c-f29c-4314-877d-15246ef14307\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-baf2cc-c47dfc746-csd9v"
Jul 29 15:15:20.279236 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.279025    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxx72\" (UniqueName: \"kubernetes.io/projected/b000ba3c-f29c-4314-877d-15246ef14307-kube-api-access-nxx72\") pod \"maas-api-e2e-api-a-baf2cc-c47dfc746-csd9v\" (UID: \"b000ba3c-f29c-4314-877d-15246ef14307\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-baf2cc-c47dfc746-csd9v"
Jul 29 15:15:20.279236 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:15:20.279158    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-baf2cc-serving-cert: secret "maas-api-e2e-api-a-baf2cc-serving-cert" not found
Jul 29 15:15:20.279236 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:15:20.279223    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b000ba3c-f29c-4314-877d-15246ef14307-maas-api-tls podName:b000ba3c-f29c-4314-877d-15246ef14307 nodeName:}" failed. No retries permitted until 2026-07-29 15:15:20.779208237 +0000 UTC m=+2422.827666154 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b000ba3c-f29c-4314-877d-15246ef14307-maas-api-tls") pod "maas-api-e2e-api-a-baf2cc-c47dfc746-csd9v" (UID: "b000ba3c-f29c-4314-877d-15246ef14307") : secret "maas-api-e2e-api-a-baf2cc-serving-cert" not found
Jul 29 15:15:20.279690 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.279672    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b000ba3c-f29c-4314-877d-15246ef14307-openshift-service-ca\") pod \"maas-api-e2e-api-a-baf2cc-c47dfc746-csd9v\" (UID: \"b000ba3c-f29c-4314-877d-15246ef14307\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-baf2cc-c47dfc746-csd9v"
Jul 29 15:15:20.288095 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.288070    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxx72\" (UniqueName: \"kubernetes.io/projected/b000ba3c-f29c-4314-877d-15246ef14307-kube-api-access-nxx72\") pod \"maas-api-e2e-api-a-baf2cc-c47dfc746-csd9v\" (UID: \"b000ba3c-f29c-4314-877d-15246ef14307\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-baf2cc-c47dfc746-csd9v"
Jul 29 15:15:20.772711 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.772683    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"
Jul 29 15:15:20.784207 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.784177    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b000ba3c-f29c-4314-877d-15246ef14307-maas-api-tls\") pod \"maas-api-e2e-api-a-baf2cc-c47dfc746-csd9v\" (UID: \"b000ba3c-f29c-4314-877d-15246ef14307\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-baf2cc-c47dfc746-csd9v"
Jul 29 15:15:20.786533 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.786513    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b000ba3c-f29c-4314-877d-15246ef14307-maas-api-tls\") pod \"maas-api-e2e-api-a-baf2cc-c47dfc746-csd9v\" (UID: \"b000ba3c-f29c-4314-877d-15246ef14307\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-baf2cc-c47dfc746-csd9v"
Jul 29 15:15:20.863438 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.863405    2563 generic.go:356] "Generic (PLEG): container finished" podID="5dbd8548-c376-46f9-925a-4f24ef7f021a" containerID="2a5f053ec2380e5d2d77e355179cf6507eeaebe78e0040da0d1e3f6828d4be09" exitCode=0
Jul 29 15:15:20.863613 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.863454    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"
Jul 29 15:15:20.863613 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.863488    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg" event={"ID":"5dbd8548-c376-46f9-925a-4f24ef7f021a","Type":"ContainerDied","Data":"2a5f053ec2380e5d2d77e355179cf6507eeaebe78e0040da0d1e3f6828d4be09"}
Jul 29 15:15:20.863613 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.863543    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg" event={"ID":"5dbd8548-c376-46f9-925a-4f24ef7f021a","Type":"ContainerDied","Data":"322df3b5e531b9dca01a2d2b9d9d6c413e9630710e738325f0614e56d01d5f34"}
Jul 29 15:15:20.863613 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.863567    2563 scope.go:117] "RemoveContainer" containerID="2a5f053ec2380e5d2d77e355179cf6507eeaebe78e0040da0d1e3f6828d4be09"
Jul 29 15:15:20.872423 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.872393    2563 scope.go:117] "RemoveContainer" containerID="6196ded0ff6446e848a022d18fa7a84fe46b09067d29472e66f595f2c779ba1c"
Jul 29 15:15:20.879971 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.879951    2563 scope.go:117] "RemoveContainer" containerID="2a5f053ec2380e5d2d77e355179cf6507eeaebe78e0040da0d1e3f6828d4be09"
Jul 29 15:15:20.880286 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:15:20.880269    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a5f053ec2380e5d2d77e355179cf6507eeaebe78e0040da0d1e3f6828d4be09\": container with ID starting with 2a5f053ec2380e5d2d77e355179cf6507eeaebe78e0040da0d1e3f6828d4be09 not found: ID does not exist" containerID="2a5f053ec2380e5d2d77e355179cf6507eeaebe78e0040da0d1e3f6828d4be09"
Jul 29 15:15:20.880354 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.880295    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a5f053ec2380e5d2d77e355179cf6507eeaebe78e0040da0d1e3f6828d4be09"} err="failed to get container status \"2a5f053ec2380e5d2d77e355179cf6507eeaebe78e0040da0d1e3f6828d4be09\": rpc error: code = NotFound desc = could not find container \"2a5f053ec2380e5d2d77e355179cf6507eeaebe78e0040da0d1e3f6828d4be09\": container with ID starting with 2a5f053ec2380e5d2d77e355179cf6507eeaebe78e0040da0d1e3f6828d4be09 not found: ID does not exist"
Jul 29 15:15:20.880354 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.880312    2563 scope.go:117] "RemoveContainer" containerID="6196ded0ff6446e848a022d18fa7a84fe46b09067d29472e66f595f2c779ba1c"
Jul 29 15:15:20.880570 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:15:20.880553    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6196ded0ff6446e848a022d18fa7a84fe46b09067d29472e66f595f2c779ba1c\": container with ID starting with 6196ded0ff6446e848a022d18fa7a84fe46b09067d29472e66f595f2c779ba1c not found: ID does not exist" containerID="6196ded0ff6446e848a022d18fa7a84fe46b09067d29472e66f595f2c779ba1c"
Jul 29 15:15:20.880612 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.880576    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6196ded0ff6446e848a022d18fa7a84fe46b09067d29472e66f595f2c779ba1c"} err="failed to get container status \"6196ded0ff6446e848a022d18fa7a84fe46b09067d29472e66f595f2c779ba1c\": rpc error: code = NotFound desc = could not find container \"6196ded0ff6446e848a022d18fa7a84fe46b09067d29472e66f595f2c779ba1c\": container with ID starting with 6196ded0ff6446e848a022d18fa7a84fe46b09067d29472e66f595f2c779ba1c not found: ID does not exist"
Jul 29 15:15:20.884870 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.884853    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dbd8548-c376-46f9-925a-4f24ef7f021a-istio-podinfo\") pod \"5dbd8548-c376-46f9-925a-4f24ef7f021a\" (UID: \"5dbd8548-c376-46f9-925a-4f24ef7f021a\") "
Jul 29 15:15:20.884930 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.884888    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbd8548-c376-46f9-925a-4f24ef7f021a-credential-socket\") pod \"5dbd8548-c376-46f9-925a-4f24ef7f021a\" (UID: \"5dbd8548-c376-46f9-925a-4f24ef7f021a\") "
Jul 29 15:15:20.884930 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.884912    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dbd8548-c376-46f9-925a-4f24ef7f021a-istio-data\") pod \"5dbd8548-c376-46f9-925a-4f24ef7f021a\" (UID: \"5dbd8548-c376-46f9-925a-4f24ef7f021a\") "
Jul 29 15:15:20.885036 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.884945    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbd8548-c376-46f9-925a-4f24ef7f021a-workload-socket\") pod \"5dbd8548-c376-46f9-925a-4f24ef7f021a\" (UID: \"5dbd8548-c376-46f9-925a-4f24ef7f021a\") "
Jul 29 15:15:20.885036 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.884982    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dbd8548-c376-46f9-925a-4f24ef7f021a-istiod-ca-cert\") pod \"5dbd8548-c376-46f9-925a-4f24ef7f021a\" (UID: \"5dbd8548-c376-46f9-925a-4f24ef7f021a\") "
Jul 29 15:15:20.885036 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.885032    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dbd8548-c376-46f9-925a-4f24ef7f021a-istio-envoy\") pod \"5dbd8548-c376-46f9-925a-4f24ef7f021a\" (UID: \"5dbd8548-c376-46f9-925a-4f24ef7f021a\") "
Jul 29 15:15:20.885201 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.885056    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49x5z\" (UniqueName: \"kubernetes.io/projected/5dbd8548-c376-46f9-925a-4f24ef7f021a-kube-api-access-49x5z\") pod \"5dbd8548-c376-46f9-925a-4f24ef7f021a\" (UID: \"5dbd8548-c376-46f9-925a-4f24ef7f021a\") "
Jul 29 15:15:20.885201 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.885081    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dbd8548-c376-46f9-925a-4f24ef7f021a-istio-token\") pod \"5dbd8548-c376-46f9-925a-4f24ef7f021a\" (UID: \"5dbd8548-c376-46f9-925a-4f24ef7f021a\") "
Jul 29 15:15:20.885201 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.885123    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dbd8548-c376-46f9-925a-4f24ef7f021a-workload-certs\") pod \"5dbd8548-c376-46f9-925a-4f24ef7f021a\" (UID: \"5dbd8548-c376-46f9-925a-4f24ef7f021a\") "
Jul 29 15:15:20.885358 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.885196    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dbd8548-c376-46f9-925a-4f24ef7f021a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5dbd8548-c376-46f9-925a-4f24ef7f021a" (UID: "5dbd8548-c376-46f9-925a-4f24ef7f021a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:15:20.885358 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.885325    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dbd8548-c376-46f9-925a-4f24ef7f021a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5dbd8548-c376-46f9-925a-4f24ef7f021a" (UID: "5dbd8548-c376-46f9-925a-4f24ef7f021a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:15:20.885487 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.885463    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbd8548-c376-46f9-925a-4f24ef7f021a-workload-socket\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:15:20.885545 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.885487    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbd8548-c376-46f9-925a-4f24ef7f021a-credential-socket\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:15:20.885545 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.885502    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dbd8548-c376-46f9-925a-4f24ef7f021a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5dbd8548-c376-46f9-925a-4f24ef7f021a" (UID: "5dbd8548-c376-46f9-925a-4f24ef7f021a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 15:15:20.885649 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.885551    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dbd8548-c376-46f9-925a-4f24ef7f021a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5dbd8548-c376-46f9-925a-4f24ef7f021a" (UID: "5dbd8548-c376-46f9-925a-4f24ef7f021a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:15:20.885879 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.885858    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dbd8548-c376-46f9-925a-4f24ef7f021a-istio-data" (OuterVolumeSpecName: "istio-data") pod "5dbd8548-c376-46f9-925a-4f24ef7f021a" (UID: "5dbd8548-c376-46f9-925a-4f24ef7f021a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:15:20.887298 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.887270    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dbd8548-c376-46f9-925a-4f24ef7f021a-istio-token" (OuterVolumeSpecName: "istio-token") pod "5dbd8548-c376-46f9-925a-4f24ef7f021a" (UID: "5dbd8548-c376-46f9-925a-4f24ef7f021a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 15:15:20.887381 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.887326    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dbd8548-c376-46f9-925a-4f24ef7f021a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5dbd8548-c376-46f9-925a-4f24ef7f021a" (UID: "5dbd8548-c376-46f9-925a-4f24ef7f021a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:15:20.887480 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.887466    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5dbd8548-c376-46f9-925a-4f24ef7f021a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5dbd8548-c376-46f9-925a-4f24ef7f021a" (UID: "5dbd8548-c376-46f9-925a-4f24ef7f021a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 15:15:20.887636 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.887621    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dbd8548-c376-46f9-925a-4f24ef7f021a-kube-api-access-49x5z" (OuterVolumeSpecName: "kube-api-access-49x5z") pod "5dbd8548-c376-46f9-925a-4f24ef7f021a" (UID: "5dbd8548-c376-46f9-925a-4f24ef7f021a"). InnerVolumeSpecName "kube-api-access-49x5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 15:15:20.941030 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.940991    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-baf2cc-c47dfc746-csd9v"
Jul 29 15:15:20.986223 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.986179    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dbd8548-c376-46f9-925a-4f24ef7f021a-workload-certs\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:15:20.986223 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.986214    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dbd8548-c376-46f9-925a-4f24ef7f021a-istio-podinfo\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:15:20.986223 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.986228    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dbd8548-c376-46f9-925a-4f24ef7f021a-istio-data\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:15:20.986521 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.986244    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dbd8548-c376-46f9-925a-4f24ef7f021a-istiod-ca-cert\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:15:20.986521 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.986256    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dbd8548-c376-46f9-925a-4f24ef7f021a-istio-envoy\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:15:20.986521 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.986268    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49x5z\" (UniqueName: \"kubernetes.io/projected/5dbd8548-c376-46f9-925a-4f24ef7f021a-kube-api-access-49x5z\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:15:20.986521 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:20.986284    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dbd8548-c376-46f9-925a-4f24ef7f021a-istio-token\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:15:21.080784 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:21.080757    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-baf2cc-c47dfc746-csd9v"]
Jul 29 15:15:21.083045 ip-10-0-136-196 kubenswrapper[2563]: W0729 15:15:21.083013    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb000ba3c_f29c_4314_877d_15246ef14307.slice/crio-f23889f62c988e4efd8e4a385aa8f799bf034f4cac05edfa0dd61b17b6a73e21 WatchSource:0}: Error finding container f23889f62c988e4efd8e4a385aa8f799bf034f4cac05edfa0dd61b17b6a73e21: Status 404 returned error can't find the container with id f23889f62c988e4efd8e4a385aa8f799bf034f4cac05edfa0dd61b17b6a73e21
Jul 29 15:15:21.084504 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:21.084485    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 15:15:21.184677 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:21.184644    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"]
Jul 29 15:15:21.189443 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:21.189420    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-80fc9d-openshift-default-84cb896c9b-nzpkg"]
Jul 29 15:15:21.870476 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:21.870439    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-baf2cc-c47dfc746-csd9v" event={"ID":"b000ba3c-f29c-4314-877d-15246ef14307","Type":"ContainerStarted","Data":"f23889f62c988e4efd8e4a385aa8f799bf034f4cac05edfa0dd61b17b6a73e21"}
Jul 29 15:15:22.437336 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:22.434089    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dbd8548-c376-46f9-925a-4f24ef7f021a" path="/var/lib/kubelet/pods/5dbd8548-c376-46f9-925a-4f24ef7f021a/volumes"
Jul 29 15:15:23.881070 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:23.881038    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-baf2cc-c47dfc746-csd9v" event={"ID":"b000ba3c-f29c-4314-877d-15246ef14307","Type":"ContainerStarted","Data":"05b32121d4e9d9ef02a37ca931bc28ddbe919007ab10ec74a70b7ee550b56ee0"}
Jul 29 15:15:23.881494 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:23.881179    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-baf2cc-c47dfc746-csd9v"
Jul 29 15:15:23.897739 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:23.897699    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-baf2cc-c47dfc746-csd9v" podStartSLOduration=1.540859459 podStartE2EDuration="3.897686981s" totalImagesPullingTime="2.356827522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:15:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:15:23.896692996 +0000 UTC m=+2425.945150949" watchObservedRunningTime="2026-07-29 15:15:23.897686981 +0000 UTC m=+2425.946144919"
Jul 29 15:15:29.891929 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:29.891836    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-baf2cc-c47dfc746-csd9v"
Jul 29 15:15:32.926993 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:32.926960    2563 generic.go:356] "Generic (PLEG): container finished" podID="d84a00f9-6797-48be-9870-75a860426818" containerID="c278cab064ee76ddc8e041d155f3d7a05ba6c5fa124ae8512441742f006083e6" exitCode=0
Jul 29 15:15:32.927440 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:32.927002    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn" event={"ID":"d84a00f9-6797-48be-9870-75a860426818","Type":"ContainerDied","Data":"c278cab064ee76ddc8e041d155f3d7a05ba6c5fa124ae8512441742f006083e6"}
Jul 29 15:15:32.927440 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:32.927379    2563 scope.go:117] "RemoveContainer" containerID="c278cab064ee76ddc8e041d155f3d7a05ba6c5fa124ae8512441742f006083e6"
Jul 29 15:15:32.928371 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:32.928342    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:15:32.928470 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:32.928410    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:15:32.928470 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:32.928443    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:15:33.852008 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:33.851908    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"
Jul 29 15:15:33.934346 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:33.934312    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn" event={"ID":"d84a00f9-6797-48be-9870-75a860426818","Type":"ContainerStarted","Data":"6b5f50dd37238c25f7686a32829375b6955cff72ed73e06c43372641aa6c400c"}
Jul 29 15:15:34.702730 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:34.702694    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"
Jul 29 15:15:34.703916 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:34.703887    2563 patch_prober.go:28] interesting pod/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Jul 29 15:15:34.704031 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:34.703945    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn" podUID="d84a00f9-6797-48be-9870-75a860426818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Jul 29 15:15:35.703029 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:35.702982    2563 patch_prober.go:28] interesting pod/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Jul 29 15:15:35.703438 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:35.703059    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn" podUID="d84a00f9-6797-48be-9870-75a860426818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Jul 29 15:15:36.702107 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:36.702070    2563 patch_prober.go:28] interesting pod/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Jul 29 15:15:36.702299 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:36.702165    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn" podUID="d84a00f9-6797-48be-9870-75a860426818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Jul 29 15:15:37.702200 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:37.702157    2563 patch_prober.go:28] interesting pod/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Jul 29 15:15:37.702689 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:37.702225    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn" podUID="d84a00f9-6797-48be-9870-75a860426818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Jul 29 15:15:38.705964 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:38.705933    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"
Jul 29 15:15:38.706449 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:38.706330    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"
Jul 29 15:15:38.706947 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:38.706930    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"
Jul 29 15:15:45.641191 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:45.641160    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"]
Jul 29 15:15:45.641547 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:45.641532    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dbd8548-c376-46f9-925a-4f24ef7f021a" containerName="istio-proxy"
Jul 29 15:15:45.641547 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:45.641543    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dbd8548-c376-46f9-925a-4f24ef7f021a" containerName="istio-proxy"
Jul 29 15:15:45.641626 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:45.641552    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dbd8548-c376-46f9-925a-4f24ef7f021a" containerName="istio-proxy"
Jul 29 15:15:45.641626 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:45.641558    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dbd8548-c376-46f9-925a-4f24ef7f021a" containerName="istio-proxy"
Jul 29 15:15:45.641626 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:45.641621    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dbd8548-c376-46f9-925a-4f24ef7f021a" containerName="istio-proxy"
Jul 29 15:15:45.641716 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:45.641632    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dbd8548-c376-46f9-925a-4f24ef7f021a" containerName="istio-proxy"
Jul 29 15:15:45.643659 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:45.643638    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"
Jul 29 15:15:45.645872 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:45.645847    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-d3e823-openshift-default-dockercfg-ww9pz\""
Jul 29 15:15:45.660402 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:45.660375    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"]
Jul 29 15:15:45.710989 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:45.710957    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs54d\" (UniqueName: \"kubernetes.io/projected/330963f5-4a33-444c-bc0c-8832a9958c3f-kube-api-access-fs54d\") pod \"e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk\" (UID: \"330963f5-4a33-444c-bc0c-8832a9958c3f\") " pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"
Jul 29 15:15:45.711126 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:45.710995    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330963f5-4a33-444c-bc0c-8832a9958c3f-istiod-ca-cert\") pod \"e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk\" (UID: \"330963f5-4a33-444c-bc0c-8832a9958c3f\") " pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"
Jul 29 15:15:45.711126 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:45.711034    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330963f5-4a33-444c-bc0c-8832a9958c3f-istio-token\") pod \"e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk\" (UID: \"330963f5-4a33-444c-bc0c-8832a9958c3f\") " pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"
Jul 29 15:15:45.711126 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:45.711050    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330963f5-4a33-444c-bc0c-8832a9958c3f-istio-envoy\") pod \"e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk\" (UID: \"330963f5-4a33-444c-bc0c-8832a9958c3f\") " pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"
Jul 29 15:15:45.711126 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:45.711067    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330963f5-4a33-444c-bc0c-8832a9958c3f-workload-certs\") pod \"e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk\" (UID: \"330963f5-4a33-444c-bc0c-8832a9958c3f\") " pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"
Jul 29 15:15:45.711126 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:45.711083    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330963f5-4a33-444c-bc0c-8832a9958c3f-credential-socket\") pod \"e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk\" (UID: \"330963f5-4a33-444c-bc0c-8832a9958c3f\") " pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"
Jul 29 15:15:45.711126 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:45.711107    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330963f5-4a33-444c-bc0c-8832a9958c3f-workload-socket\") pod \"e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk\" (UID: \"330963f5-4a33-444c-bc0c-8832a9958c3f\") " pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"
Jul 29 15:15:45.711357 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:45.711199    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330963f5-4a33-444c-bc0c-8832a9958c3f-istio-data\") pod \"e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk\" (UID: \"330963f5-4a33-444c-bc0c-8832a9958c3f\") " pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"
Jul 29 15:15:45.711357 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:45.711228    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330963f5-4a33-444c-bc0c-8832a9958c3f-istio-podinfo\") pod \"e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk\" (UID: \"330963f5-4a33-444c-bc0c-8832a9958c3f\") " pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"
Jul 29 15:15:45.812819 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:45.812780    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330963f5-4a33-444c-bc0c-8832a9958c3f-istiod-ca-cert\") pod \"e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk\" (UID: \"330963f5-4a33-444c-bc0c-8832a9958c3f\") " pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"
Jul 29 15:15:45.813002 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:45.812867    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330963f5-4a33-444c-bc0c-8832a9958c3f-istio-token\") pod \"e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk\" (UID: \"330963f5-4a33-444c-bc0c-8832a9958c3f\") " pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"
Jul 29 15:15:45.813002 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:45.812899    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330963f5-4a33-444c-bc0c-8832a9958c3f-istio-envoy\") pod \"e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk\" (UID: \"330963f5-4a33-444c-bc0c-8832a9958c3f\") " pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"
Jul 29 15:15:45.813002 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:45.812928    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330963f5-4a33-444c-bc0c-8832a9958c3f-workload-certs\") pod \"e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk\" (UID: \"330963f5-4a33-444c-bc0c-8832a9958c3f\") " pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"
Jul 29 15:15:45.813214 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:45.813126    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330963f5-4a33-444c-bc0c-8832a9958c3f-credential-socket\") pod \"e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk\" (UID: \"330963f5-4a33-444c-bc0c-8832a9958c3f\") " pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"
Jul 29 15:15:45.813271 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:45.813223    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330963f5-4a33-444c-bc0c-8832a9958c3f-workload-socket\") pod \"e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk\" (UID: \"330963f5-4a33-444c-bc0c-8832a9958c3f\") " pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"
Jul 29 15:15:45.813322 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:45.813269    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330963f5-4a33-444c-bc0c-8832a9958c3f-istio-data\") pod \"e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk\" (UID: \"330963f5-4a33-444c-bc0c-8832a9958c3f\") " pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"
Jul 29 15:15:45.813322 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:45.813281    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330963f5-4a33-444c-bc0c-8832a9958c3f-workload-certs\") pod \"e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk\" (UID: \"330963f5-4a33-444c-bc0c-8832a9958c3f\") " pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"
Jul 29 15:15:45.813322 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:45.813297    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330963f5-4a33-444c-bc0c-8832a9958c3f-istio-podinfo\") pod \"e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk\" (UID: \"330963f5-4a33-444c-bc0c-8832a9958c3f\") " pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"
Jul 29 15:15:45.813536 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:45.813349    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs54d\" (UniqueName: \"kubernetes.io/projected/330963f5-4a33-444c-bc0c-8832a9958c3f-kube-api-access-fs54d\") pod \"e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk\" (UID: \"330963f5-4a33-444c-bc0c-8832a9958c3f\") " pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"
Jul 29 15:15:45.813536 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:45.813473    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330963f5-4a33-444c-bc0c-8832a9958c3f-credential-socket\") pod \"e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk\" (UID: \"330963f5-4a33-444c-bc0c-8832a9958c3f\") " pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"
Jul 29 15:15:45.813536 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:45.813519    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330963f5-4a33-444c-bc0c-8832a9958c3f-workload-socket\") pod \"e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk\" (UID: \"330963f5-4a33-444c-bc0c-8832a9958c3f\") " pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"
Jul 29 15:15:45.813702 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:45.813607    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330963f5-4a33-444c-bc0c-8832a9958c3f-istiod-ca-cert\") pod \"e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk\" (UID: \"330963f5-4a33-444c-bc0c-8832a9958c3f\") " pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"
Jul 29 15:15:45.813702 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:45.813674    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330963f5-4a33-444c-bc0c-8832a9958c3f-istio-data\") pod \"e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk\" (UID: \"330963f5-4a33-444c-bc0c-8832a9958c3f\") " pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"
Jul 29 15:15:45.815420 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:45.815395    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330963f5-4a33-444c-bc0c-8832a9958c3f-istio-envoy\") pod \"e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk\" (UID: \"330963f5-4a33-444c-bc0c-8832a9958c3f\") " pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"
Jul 29 15:15:45.815831 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:45.815809    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330963f5-4a33-444c-bc0c-8832a9958c3f-istio-podinfo\") pod \"e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk\" (UID: \"330963f5-4a33-444c-bc0c-8832a9958c3f\") " pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"
Jul 29 15:15:45.820941 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:45.820916    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330963f5-4a33-444c-bc0c-8832a9958c3f-istio-token\") pod \"e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk\" (UID: \"330963f5-4a33-444c-bc0c-8832a9958c3f\") " pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"
Jul 29 15:15:45.821278 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:45.821254    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs54d\" (UniqueName: \"kubernetes.io/projected/330963f5-4a33-444c-bc0c-8832a9958c3f-kube-api-access-fs54d\") pod \"e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk\" (UID: \"330963f5-4a33-444c-bc0c-8832a9958c3f\") " pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"
Jul 29 15:15:45.955072 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:45.955028    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"
Jul 29 15:15:46.095580 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:46.095552    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"]
Jul 29 15:15:46.097680 ip-10-0-136-196 kubenswrapper[2563]: W0729 15:15:46.097652    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330963f5_4a33_444c_bc0c_8832a9958c3f.slice/crio-bb405ea55bc862ffc7f51c8295fac8d8b068c408801235f2cfd61861b0a7fd4e WatchSource:0}: Error finding container bb405ea55bc862ffc7f51c8295fac8d8b068c408801235f2cfd61861b0a7fd4e: Status 404 returned error can't find the container with id bb405ea55bc862ffc7f51c8295fac8d8b068c408801235f2cfd61861b0a7fd4e
Jul 29 15:15:46.100368 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:46.100334    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:15:46.100464 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:46.100405    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:15:46.100464 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:46.100432    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:15:47.002439 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:47.002406    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk" event={"ID":"330963f5-4a33-444c-bc0c-8832a9958c3f","Type":"ContainerStarted","Data":"feb0a563037f7556bb7a9750cf5afe76f4eca9339922265d5da38cc9e7b1af64"}
Jul 29 15:15:47.002813 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:47.002461    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk" event={"ID":"330963f5-4a33-444c-bc0c-8832a9958c3f","Type":"ContainerStarted","Data":"bb405ea55bc862ffc7f51c8295fac8d8b068c408801235f2cfd61861b0a7fd4e"}
Jul 29 15:15:47.029281 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:47.029234    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk" podStartSLOduration=2.029218061 podStartE2EDuration="2.029218061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:15:47.027215754 +0000 UTC m=+2449.075673692" watchObservedRunningTime="2026-07-29 15:15:47.029218061 +0000 UTC m=+2449.077676002"
Jul 29 15:15:47.955436 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:47.955402    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"
Jul 29 15:15:47.960267 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:47.960240    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"
Jul 29 15:15:48.006948 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:48.006916    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"
Jul 29 15:15:48.007998 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:48.007978    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"
Jul 29 15:15:49.035011 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:49.034967    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d3e823-857fbbcbf7-sqpvd"]
Jul 29 15:15:49.038056 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:49.038033    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3e823-857fbbcbf7-sqpvd"
Jul 29 15:15:49.040077 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:49.040055    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-d3e823-serving-cert\""
Jul 29 15:15:49.050467 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:49.050434    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d3e823-857fbbcbf7-sqpvd"]
Jul 29 15:15:49.149958 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:49.149916    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/042d3379-8772-400f-b264-b8597b7e0bc2-openshift-service-ca\") pod \"maas-api-e2e-api-b-d3e823-857fbbcbf7-sqpvd\" (UID: \"042d3379-8772-400f-b264-b8597b7e0bc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3e823-857fbbcbf7-sqpvd"
Jul 29 15:15:49.150180 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:49.149978    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rznrh\" (UniqueName: \"kubernetes.io/projected/042d3379-8772-400f-b264-b8597b7e0bc2-kube-api-access-rznrh\") pod \"maas-api-e2e-api-b-d3e823-857fbbcbf7-sqpvd\" (UID: \"042d3379-8772-400f-b264-b8597b7e0bc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3e823-857fbbcbf7-sqpvd"
Jul 29 15:15:49.150180 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:49.150086    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/042d3379-8772-400f-b264-b8597b7e0bc2-maas-api-tls\") pod \"maas-api-e2e-api-b-d3e823-857fbbcbf7-sqpvd\" (UID: \"042d3379-8772-400f-b264-b8597b7e0bc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3e823-857fbbcbf7-sqpvd"
Jul 29 15:15:49.251448 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:49.251413    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/042d3379-8772-400f-b264-b8597b7e0bc2-maas-api-tls\") pod \"maas-api-e2e-api-b-d3e823-857fbbcbf7-sqpvd\" (UID: \"042d3379-8772-400f-b264-b8597b7e0bc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3e823-857fbbcbf7-sqpvd"
Jul 29 15:15:49.251624 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:49.251466    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/042d3379-8772-400f-b264-b8597b7e0bc2-openshift-service-ca\") pod \"maas-api-e2e-api-b-d3e823-857fbbcbf7-sqpvd\" (UID: \"042d3379-8772-400f-b264-b8597b7e0bc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3e823-857fbbcbf7-sqpvd"
Jul 29 15:15:49.251624 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:49.251488    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rznrh\" (UniqueName: \"kubernetes.io/projected/042d3379-8772-400f-b264-b8597b7e0bc2-kube-api-access-rznrh\") pod \"maas-api-e2e-api-b-d3e823-857fbbcbf7-sqpvd\" (UID: \"042d3379-8772-400f-b264-b8597b7e0bc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3e823-857fbbcbf7-sqpvd"
Jul 29 15:15:49.251624 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:15:49.251561    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-d3e823-serving-cert: secret "maas-api-e2e-api-b-d3e823-serving-cert" not found
Jul 29 15:15:49.251624 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:15:49.251625    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/042d3379-8772-400f-b264-b8597b7e0bc2-maas-api-tls podName:042d3379-8772-400f-b264-b8597b7e0bc2 nodeName:}" failed. No retries permitted until 2026-07-29 15:15:49.751609838 +0000 UTC m=+2451.800067754 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/042d3379-8772-400f-b264-b8597b7e0bc2-maas-api-tls") pod "maas-api-e2e-api-b-d3e823-857fbbcbf7-sqpvd" (UID: "042d3379-8772-400f-b264-b8597b7e0bc2") : secret "maas-api-e2e-api-b-d3e823-serving-cert" not found
Jul 29 15:15:49.252161 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:49.252122    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/042d3379-8772-400f-b264-b8597b7e0bc2-openshift-service-ca\") pod \"maas-api-e2e-api-b-d3e823-857fbbcbf7-sqpvd\" (UID: \"042d3379-8772-400f-b264-b8597b7e0bc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3e823-857fbbcbf7-sqpvd"
Jul 29 15:15:49.260054 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:49.260034    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rznrh\" (UniqueName: \"kubernetes.io/projected/042d3379-8772-400f-b264-b8597b7e0bc2-kube-api-access-rznrh\") pod \"maas-api-e2e-api-b-d3e823-857fbbcbf7-sqpvd\" (UID: \"042d3379-8772-400f-b264-b8597b7e0bc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3e823-857fbbcbf7-sqpvd"
Jul 29 15:15:49.756396 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:49.756362    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/042d3379-8772-400f-b264-b8597b7e0bc2-maas-api-tls\") pod \"maas-api-e2e-api-b-d3e823-857fbbcbf7-sqpvd\" (UID: \"042d3379-8772-400f-b264-b8597b7e0bc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3e823-857fbbcbf7-sqpvd"
Jul 29 15:15:49.758961 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:49.758929    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/042d3379-8772-400f-b264-b8597b7e0bc2-maas-api-tls\") pod \"maas-api-e2e-api-b-d3e823-857fbbcbf7-sqpvd\" (UID: \"042d3379-8772-400f-b264-b8597b7e0bc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3e823-857fbbcbf7-sqpvd"
Jul 29 15:15:49.949744 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:49.949710    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3e823-857fbbcbf7-sqpvd"
Jul 29 15:15:50.087079 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:50.087023    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d3e823-857fbbcbf7-sqpvd"]
Jul 29 15:15:50.090231 ip-10-0-136-196 kubenswrapper[2563]: W0729 15:15:50.090204    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod042d3379_8772_400f_b264_b8597b7e0bc2.slice/crio-9f6803d21d0f8da0708903e59e7b04fb28ff0d9244e91826d7d6230f475180f7 WatchSource:0}: Error finding container 9f6803d21d0f8da0708903e59e7b04fb28ff0d9244e91826d7d6230f475180f7: Status 404 returned error can't find the container with id 9f6803d21d0f8da0708903e59e7b04fb28ff0d9244e91826d7d6230f475180f7
Jul 29 15:15:51.023484 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:51.023448    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3e823-857fbbcbf7-sqpvd" event={"ID":"042d3379-8772-400f-b264-b8597b7e0bc2","Type":"ContainerStarted","Data":"9e975b2744a8e7d0bd09d72ce8723638d0bbc352bf558fa93a2608b3fa495456"}
Jul 29 15:15:51.023696 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:51.023501    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3e823-857fbbcbf7-sqpvd" event={"ID":"042d3379-8772-400f-b264-b8597b7e0bc2","Type":"ContainerStarted","Data":"9f6803d21d0f8da0708903e59e7b04fb28ff0d9244e91826d7d6230f475180f7"}
Jul 29 15:15:51.023696 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:51.023547    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3e823-857fbbcbf7-sqpvd"
Jul 29 15:15:51.040525 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:51.040471    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3e823-857fbbcbf7-sqpvd" podStartSLOduration=1.669128335 podStartE2EDuration="2.040454498s" totalImagesPullingTime="371.326163ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:15:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:15:51.040060757 +0000 UTC m=+2453.088518697" watchObservedRunningTime="2026-07-29 15:15:51.040454498 +0000 UTC m=+2453.088912437"
Jul 29 15:15:57.034565 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:15:57.034526    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3e823-857fbbcbf7-sqpvd"
Jul 29 15:16:04.049327 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:04.049279    2563 patch_prober.go:28] interesting pod/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 29 15:16:04.049728 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:04.049361    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk" podUID="330963f5-4a33-444c-bc0c-8832a9958c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 29 15:16:05.104694 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:05.104658    2563 generic.go:356] "Generic (PLEG): container finished" podID="330963f5-4a33-444c-bc0c-8832a9958c3f" containerID="feb0a563037f7556bb7a9750cf5afe76f4eca9339922265d5da38cc9e7b1af64" exitCode=0
Jul 29 15:16:05.105306 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:05.104734    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk" event={"ID":"330963f5-4a33-444c-bc0c-8832a9958c3f","Type":"ContainerDied","Data":"feb0a563037f7556bb7a9750cf5afe76f4eca9339922265d5da38cc9e7b1af64"}
Jul 29 15:16:05.105306 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:05.105228    2563 scope.go:117] "RemoveContainer" containerID="feb0a563037f7556bb7a9750cf5afe76f4eca9339922265d5da38cc9e7b1af64"
Jul 29 15:16:05.106309 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:05.106283    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:16:05.106411 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:05.106348    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:16:05.106411 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:05.106385    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:16:06.111381 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:06.111345    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk" event={"ID":"330963f5-4a33-444c-bc0c-8832a9958c3f","Type":"ContainerStarted","Data":"a9c03f15f5acebfed0ccf212a23185ca8cc6d850c5f602462037bf4aaac69f15"}
Jul 29 15:16:06.955421 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:06.955383    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"
Jul 29 15:16:06.955695 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:06.955662    2563 patch_prober.go:28] interesting pod/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Jul 29 15:16:06.955905 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:06.955876    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk" podUID="330963f5-4a33-444c-bc0c-8832a9958c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Jul 29 15:16:07.955713 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:07.955668    2563 patch_prober.go:28] interesting pod/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Jul 29 15:16:07.956224 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:07.955735    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk" podUID="330963f5-4a33-444c-bc0c-8832a9958c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Jul 29 15:16:08.955513 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:08.955467    2563 patch_prober.go:28] interesting pod/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Jul 29 15:16:08.955727 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:08.955536    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk" podUID="330963f5-4a33-444c-bc0c-8832a9958c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Jul 29 15:16:09.959294 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:09.959262    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"
Jul 29 15:16:09.959766 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:09.959660    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"
Jul 29 15:16:09.960423 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:09.960400    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"
Jul 29 15:16:32.699419 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:32.699381    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-baf2cc-c47dfc746-csd9v"]
Jul 29 15:16:32.699816 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:32.699645    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-baf2cc-c47dfc746-csd9v" podUID="b000ba3c-f29c-4314-877d-15246ef14307" containerName="maas-api" containerID="cri-o://05b32121d4e9d9ef02a37ca931bc28ddbe919007ab10ec74a70b7ee550b56ee0" gracePeriod=30
Jul 29 15:16:32.957873 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:32.957847    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-baf2cc-c47dfc746-csd9v"
Jul 29 15:16:33.051672 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:33.051640    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b000ba3c-f29c-4314-877d-15246ef14307-openshift-service-ca\") pod \"b000ba3c-f29c-4314-877d-15246ef14307\" (UID: \"b000ba3c-f29c-4314-877d-15246ef14307\") "
Jul 29 15:16:33.051866 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:33.051685    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b000ba3c-f29c-4314-877d-15246ef14307-maas-api-tls\") pod \"b000ba3c-f29c-4314-877d-15246ef14307\" (UID: \"b000ba3c-f29c-4314-877d-15246ef14307\") "
Jul 29 15:16:33.051866 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:33.051736    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxx72\" (UniqueName: \"kubernetes.io/projected/b000ba3c-f29c-4314-877d-15246ef14307-kube-api-access-nxx72\") pod \"b000ba3c-f29c-4314-877d-15246ef14307\" (UID: \"b000ba3c-f29c-4314-877d-15246ef14307\") "
Jul 29 15:16:33.052069 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:33.052045    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b000ba3c-f29c-4314-877d-15246ef14307-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b000ba3c-f29c-4314-877d-15246ef14307" (UID: "b000ba3c-f29c-4314-877d-15246ef14307"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 15:16:33.053912 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:33.053879    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b000ba3c-f29c-4314-877d-15246ef14307-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b000ba3c-f29c-4314-877d-15246ef14307" (UID: "b000ba3c-f29c-4314-877d-15246ef14307"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 15:16:33.054057 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:33.053917    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b000ba3c-f29c-4314-877d-15246ef14307-kube-api-access-nxx72" (OuterVolumeSpecName: "kube-api-access-nxx72") pod "b000ba3c-f29c-4314-877d-15246ef14307" (UID: "b000ba3c-f29c-4314-877d-15246ef14307"). InnerVolumeSpecName "kube-api-access-nxx72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 15:16:33.152390 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:33.152357    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxx72\" (UniqueName: \"kubernetes.io/projected/b000ba3c-f29c-4314-877d-15246ef14307-kube-api-access-nxx72\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:16:33.152390 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:33.152387    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b000ba3c-f29c-4314-877d-15246ef14307-openshift-service-ca\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:16:33.152390 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:33.152398    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b000ba3c-f29c-4314-877d-15246ef14307-maas-api-tls\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:16:33.249701 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:33.249609    2563 generic.go:356] "Generic (PLEG): container finished" podID="b000ba3c-f29c-4314-877d-15246ef14307" containerID="05b32121d4e9d9ef02a37ca931bc28ddbe919007ab10ec74a70b7ee550b56ee0" exitCode=0
Jul 29 15:16:33.249701 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:33.249654    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-baf2cc-c47dfc746-csd9v"
Jul 29 15:16:33.249951 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:33.249698    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-baf2cc-c47dfc746-csd9v" event={"ID":"b000ba3c-f29c-4314-877d-15246ef14307","Type":"ContainerDied","Data":"05b32121d4e9d9ef02a37ca931bc28ddbe919007ab10ec74a70b7ee550b56ee0"}
Jul 29 15:16:33.249951 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:33.249756    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-baf2cc-c47dfc746-csd9v" event={"ID":"b000ba3c-f29c-4314-877d-15246ef14307","Type":"ContainerDied","Data":"f23889f62c988e4efd8e4a385aa8f799bf034f4cac05edfa0dd61b17b6a73e21"}
Jul 29 15:16:33.249951 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:33.249787    2563 scope.go:117] "RemoveContainer" containerID="05b32121d4e9d9ef02a37ca931bc28ddbe919007ab10ec74a70b7ee550b56ee0"
Jul 29 15:16:33.260354 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:33.260279    2563 scope.go:117] "RemoveContainer" containerID="05b32121d4e9d9ef02a37ca931bc28ddbe919007ab10ec74a70b7ee550b56ee0"
Jul 29 15:16:33.260795 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:16:33.260774    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05b32121d4e9d9ef02a37ca931bc28ddbe919007ab10ec74a70b7ee550b56ee0\": container with ID starting with 05b32121d4e9d9ef02a37ca931bc28ddbe919007ab10ec74a70b7ee550b56ee0 not found: ID does not exist" containerID="05b32121d4e9d9ef02a37ca931bc28ddbe919007ab10ec74a70b7ee550b56ee0"
Jul 29 15:16:33.260905 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:33.260804    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05b32121d4e9d9ef02a37ca931bc28ddbe919007ab10ec74a70b7ee550b56ee0"} err="failed to get container status \"05b32121d4e9d9ef02a37ca931bc28ddbe919007ab10ec74a70b7ee550b56ee0\": rpc error: code = NotFound desc = could not find container \"05b32121d4e9d9ef02a37ca931bc28ddbe919007ab10ec74a70b7ee550b56ee0\": container with ID starting with 05b32121d4e9d9ef02a37ca931bc28ddbe919007ab10ec74a70b7ee550b56ee0 not found: ID does not exist"
Jul 29 15:16:33.278125 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:33.278102    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-baf2cc-c47dfc746-csd9v"]
Jul 29 15:16:33.282013 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:33.281992    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-baf2cc-c47dfc746-csd9v"]
Jul 29 15:16:34.432437 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:34.432403    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b000ba3c-f29c-4314-877d-15246ef14307" path="/var/lib/kubelet/pods/b000ba3c-f29c-4314-877d-15246ef14307/volumes"
Jul 29 15:16:39.170319 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:39.170278    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"]
Jul 29 15:16:39.170880 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:39.170547    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn" podUID="d84a00f9-6797-48be-9870-75a860426818" containerName="istio-proxy" containerID="cri-o://6b5f50dd37238c25f7686a32829375b6955cff72ed73e06c43372641aa6c400c" gracePeriod=30
Jul 29 15:16:44.308163 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:44.308116    2563 generic.go:356] "Generic (PLEG): container finished" podID="d84a00f9-6797-48be-9870-75a860426818" containerID="6b5f50dd37238c25f7686a32829375b6955cff72ed73e06c43372641aa6c400c" exitCode=0
Jul 29 15:16:44.308539 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:44.308170    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn" event={"ID":"d84a00f9-6797-48be-9870-75a860426818","Type":"ContainerDied","Data":"6b5f50dd37238c25f7686a32829375b6955cff72ed73e06c43372641aa6c400c"}
Jul 29 15:16:44.308539 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:44.308241    2563 scope.go:117] "RemoveContainer" containerID="c278cab064ee76ddc8e041d155f3d7a05ba6c5fa124ae8512441742f006083e6"
Jul 29 15:16:44.423970 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:44.423945    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"
Jul 29 15:16:44.457103 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:44.457078    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d84a00f9-6797-48be-9870-75a860426818-workload-certs\") pod \"d84a00f9-6797-48be-9870-75a860426818\" (UID: \"d84a00f9-6797-48be-9870-75a860426818\") "
Jul 29 15:16:44.457352 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:44.457123    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d84a00f9-6797-48be-9870-75a860426818-istio-podinfo\") pod \"d84a00f9-6797-48be-9870-75a860426818\" (UID: \"d84a00f9-6797-48be-9870-75a860426818\") "
Jul 29 15:16:44.457352 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:44.457180    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84a00f9-6797-48be-9870-75a860426818-credential-socket\") pod \"d84a00f9-6797-48be-9870-75a860426818\" (UID: \"d84a00f9-6797-48be-9870-75a860426818\") "
Jul 29 15:16:44.457352 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:44.457201    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84a00f9-6797-48be-9870-75a860426818-workload-socket\") pod \"d84a00f9-6797-48be-9870-75a860426818\" (UID: \"d84a00f9-6797-48be-9870-75a860426818\") "
Jul 29 15:16:44.457352 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:44.457228    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d84a00f9-6797-48be-9870-75a860426818-istio-envoy\") pod \"d84a00f9-6797-48be-9870-75a860426818\" (UID: \"d84a00f9-6797-48be-9870-75a860426818\") "
Jul 29 15:16:44.457352 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:44.457250    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plkqc\" (UniqueName: \"kubernetes.io/projected/d84a00f9-6797-48be-9870-75a860426818-kube-api-access-plkqc\") pod \"d84a00f9-6797-48be-9870-75a860426818\" (UID: \"d84a00f9-6797-48be-9870-75a860426818\") "
Jul 29 15:16:44.457352 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:44.457322    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d84a00f9-6797-48be-9870-75a860426818-istiod-ca-cert\") pod \"d84a00f9-6797-48be-9870-75a860426818\" (UID: \"d84a00f9-6797-48be-9870-75a860426818\") "
Jul 29 15:16:44.457669 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:44.457370    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84a00f9-6797-48be-9870-75a860426818-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d84a00f9-6797-48be-9870-75a860426818" (UID: "d84a00f9-6797-48be-9870-75a860426818"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:16:44.457669 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:44.457379    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d84a00f9-6797-48be-9870-75a860426818-istio-data\") pod \"d84a00f9-6797-48be-9870-75a860426818\" (UID: \"d84a00f9-6797-48be-9870-75a860426818\") "
Jul 29 15:16:44.457669 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:44.457511    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84a00f9-6797-48be-9870-75a860426818-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d84a00f9-6797-48be-9870-75a860426818" (UID: "d84a00f9-6797-48be-9870-75a860426818"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:16:44.457669 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:44.457588    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84a00f9-6797-48be-9870-75a860426818-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d84a00f9-6797-48be-9870-75a860426818" (UID: "d84a00f9-6797-48be-9870-75a860426818"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:16:44.457887 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:44.457777    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d84a00f9-6797-48be-9870-75a860426818-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d84a00f9-6797-48be-9870-75a860426818" (UID: "d84a00f9-6797-48be-9870-75a860426818"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 15:16:44.457962 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:44.457906    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d84a00f9-6797-48be-9870-75a860426818-istio-token\") pod \"d84a00f9-6797-48be-9870-75a860426818\" (UID: \"d84a00f9-6797-48be-9870-75a860426818\") "
Jul 29 15:16:44.458366 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:44.458343    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84a00f9-6797-48be-9870-75a860426818-credential-socket\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:16:44.458486 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:44.458370    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84a00f9-6797-48be-9870-75a860426818-workload-socket\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:16:44.458486 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:44.458398    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d84a00f9-6797-48be-9870-75a860426818-istiod-ca-cert\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:16:44.458486 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:44.458413    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d84a00f9-6797-48be-9870-75a860426818-workload-certs\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:16:44.458677 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:44.458654    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84a00f9-6797-48be-9870-75a860426818-istio-data" (OuterVolumeSpecName: "istio-data") pod "d84a00f9-6797-48be-9870-75a860426818" (UID: "d84a00f9-6797-48be-9870-75a860426818"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:16:44.459583 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:44.459559    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d84a00f9-6797-48be-9870-75a860426818-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d84a00f9-6797-48be-9870-75a860426818" (UID: "d84a00f9-6797-48be-9870-75a860426818"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 15:16:44.459707 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:44.459620    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84a00f9-6797-48be-9870-75a860426818-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d84a00f9-6797-48be-9870-75a860426818" (UID: "d84a00f9-6797-48be-9870-75a860426818"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:16:44.459768 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:44.459750    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d84a00f9-6797-48be-9870-75a860426818-kube-api-access-plkqc" (OuterVolumeSpecName: "kube-api-access-plkqc") pod "d84a00f9-6797-48be-9870-75a860426818" (UID: "d84a00f9-6797-48be-9870-75a860426818"). InnerVolumeSpecName "kube-api-access-plkqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 15:16:44.459959 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:44.459938    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d84a00f9-6797-48be-9870-75a860426818-istio-token" (OuterVolumeSpecName: "istio-token") pod "d84a00f9-6797-48be-9870-75a860426818" (UID: "d84a00f9-6797-48be-9870-75a860426818"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 15:16:44.559740 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:44.559704    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d84a00f9-6797-48be-9870-75a860426818-istio-envoy\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:16:44.559740 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:44.559736    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plkqc\" (UniqueName: \"kubernetes.io/projected/d84a00f9-6797-48be-9870-75a860426818-kube-api-access-plkqc\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:16:44.559740 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:44.559747    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d84a00f9-6797-48be-9870-75a860426818-istio-data\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:16:44.560004 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:44.559756    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d84a00f9-6797-48be-9870-75a860426818-istio-token\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:16:44.560004 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:44.559763    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d84a00f9-6797-48be-9870-75a860426818-istio-podinfo\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:16:45.314278 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:45.314240    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn" event={"ID":"d84a00f9-6797-48be-9870-75a860426818","Type":"ContainerDied","Data":"d45bdcd7083b2d9356cea5559796ac0b09de7d60b26ec62c7861e7c2298e125b"}
Jul 29 15:16:45.314278 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:45.314265    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"
Jul 29 15:16:45.314868 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:45.314313    2563 scope.go:117] "RemoveContainer" containerID="6b5f50dd37238c25f7686a32829375b6955cff72ed73e06c43372641aa6c400c"
Jul 29 15:16:45.337238 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:45.337209    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"]
Jul 29 15:16:45.342771 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:45.342748    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-baf2cc-openshift-default-67c67dd4cd-xrxhn"]
Jul 29 15:16:46.438808 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:46.438769    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d84a00f9-6797-48be-9870-75a860426818" path="/var/lib/kubelet/pods/d84a00f9-6797-48be-9870-75a860426818/volumes"
Jul 29 15:16:55.110788 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:55.110696    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d3e823-857fbbcbf7-sqpvd"]
Jul 29 15:16:55.111264 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:55.110934    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3e823-857fbbcbf7-sqpvd" podUID="042d3379-8772-400f-b264-b8597b7e0bc2" containerName="maas-api" containerID="cri-o://9e975b2744a8e7d0bd09d72ce8723638d0bbc352bf558fa93a2608b3fa495456" gracePeriod=30
Jul 29 15:16:55.366205 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:55.366143    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3e823-857fbbcbf7-sqpvd"
Jul 29 15:16:55.371575 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:55.371547    2563 generic.go:356] "Generic (PLEG): container finished" podID="042d3379-8772-400f-b264-b8597b7e0bc2" containerID="9e975b2744a8e7d0bd09d72ce8723638d0bbc352bf558fa93a2608b3fa495456" exitCode=0
Jul 29 15:16:55.371688 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:55.371598    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3e823-857fbbcbf7-sqpvd"
Jul 29 15:16:55.371688 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:55.371638    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3e823-857fbbcbf7-sqpvd" event={"ID":"042d3379-8772-400f-b264-b8597b7e0bc2","Type":"ContainerDied","Data":"9e975b2744a8e7d0bd09d72ce8723638d0bbc352bf558fa93a2608b3fa495456"}
Jul 29 15:16:55.371762 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:55.371690    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3e823-857fbbcbf7-sqpvd" event={"ID":"042d3379-8772-400f-b264-b8597b7e0bc2","Type":"ContainerDied","Data":"9f6803d21d0f8da0708903e59e7b04fb28ff0d9244e91826d7d6230f475180f7"}
Jul 29 15:16:55.371762 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:55.371714    2563 scope.go:117] "RemoveContainer" containerID="9e975b2744a8e7d0bd09d72ce8723638d0bbc352bf558fa93a2608b3fa495456"
Jul 29 15:16:55.381322 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:55.381277    2563 scope.go:117] "RemoveContainer" containerID="9e975b2744a8e7d0bd09d72ce8723638d0bbc352bf558fa93a2608b3fa495456"
Jul 29 15:16:55.381721 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:16:55.381691    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e975b2744a8e7d0bd09d72ce8723638d0bbc352bf558fa93a2608b3fa495456\": container with ID starting with 9e975b2744a8e7d0bd09d72ce8723638d0bbc352bf558fa93a2608b3fa495456 not found: ID does not exist" containerID="9e975b2744a8e7d0bd09d72ce8723638d0bbc352bf558fa93a2608b3fa495456"
Jul 29 15:16:55.381845 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:55.381732    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e975b2744a8e7d0bd09d72ce8723638d0bbc352bf558fa93a2608b3fa495456"} err="failed to get container status \"9e975b2744a8e7d0bd09d72ce8723638d0bbc352bf558fa93a2608b3fa495456\": rpc error: code = NotFound desc = could not find container \"9e975b2744a8e7d0bd09d72ce8723638d0bbc352bf558fa93a2608b3fa495456\": container with ID starting with 9e975b2744a8e7d0bd09d72ce8723638d0bbc352bf558fa93a2608b3fa495456 not found: ID does not exist"
Jul 29 15:16:55.460032 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:55.459998    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/042d3379-8772-400f-b264-b8597b7e0bc2-openshift-service-ca\") pod \"042d3379-8772-400f-b264-b8597b7e0bc2\" (UID: \"042d3379-8772-400f-b264-b8597b7e0bc2\") "
Jul 29 15:16:55.460229 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:55.460097    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rznrh\" (UniqueName: \"kubernetes.io/projected/042d3379-8772-400f-b264-b8597b7e0bc2-kube-api-access-rznrh\") pod \"042d3379-8772-400f-b264-b8597b7e0bc2\" (UID: \"042d3379-8772-400f-b264-b8597b7e0bc2\") "
Jul 29 15:16:55.460229 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:55.460192    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/042d3379-8772-400f-b264-b8597b7e0bc2-maas-api-tls\") pod \"042d3379-8772-400f-b264-b8597b7e0bc2\" (UID: \"042d3379-8772-400f-b264-b8597b7e0bc2\") "
Jul 29 15:16:55.460495 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:55.460461    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/042d3379-8772-400f-b264-b8597b7e0bc2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "042d3379-8772-400f-b264-b8597b7e0bc2" (UID: "042d3379-8772-400f-b264-b8597b7e0bc2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 15:16:55.462275 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:55.462251    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/042d3379-8772-400f-b264-b8597b7e0bc2-kube-api-access-rznrh" (OuterVolumeSpecName: "kube-api-access-rznrh") pod "042d3379-8772-400f-b264-b8597b7e0bc2" (UID: "042d3379-8772-400f-b264-b8597b7e0bc2"). InnerVolumeSpecName "kube-api-access-rznrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 15:16:55.462412 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:55.462268    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/042d3379-8772-400f-b264-b8597b7e0bc2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "042d3379-8772-400f-b264-b8597b7e0bc2" (UID: "042d3379-8772-400f-b264-b8597b7e0bc2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 15:16:55.561536 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:55.561502    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/042d3379-8772-400f-b264-b8597b7e0bc2-openshift-service-ca\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:16:55.561536 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:55.561530    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rznrh\" (UniqueName: \"kubernetes.io/projected/042d3379-8772-400f-b264-b8597b7e0bc2-kube-api-access-rznrh\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:16:55.561536 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:55.561542    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/042d3379-8772-400f-b264-b8597b7e0bc2-maas-api-tls\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:16:55.692604 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:55.692566    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d3e823-857fbbcbf7-sqpvd"]
Jul 29 15:16:55.696103 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:55.696074    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d3e823-857fbbcbf7-sqpvd"]
Jul 29 15:16:56.431780 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:56.431749    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="042d3379-8772-400f-b264-b8597b7e0bc2" path="/var/lib/kubelet/pods/042d3379-8772-400f-b264-b8597b7e0bc2/volumes"
Jul 29 15:16:58.947985 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:16:58.947950    2563 scope.go:117] "RemoveContainer" containerID="a812e61ff3a85d566436b396252ba096b5e884f2bb524f5b2535b4a7ac338242"
Jul 29 15:17:01.932449 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:01.932409    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"]
Jul 29 15:17:01.932933 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:01.932686    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk" podUID="330963f5-4a33-444c-bc0c-8832a9958c3f" containerName="istio-proxy" containerID="cri-o://a9c03f15f5acebfed0ccf212a23185ca8cc6d850c5f602462037bf4aaac69f15" gracePeriod=30
Jul 29 15:17:05.431272 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:05.431238    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c73214-6b499bc688-pc4tt"]
Jul 29 15:17:05.431749 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:05.431600    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b000ba3c-f29c-4314-877d-15246ef14307" containerName="maas-api"
Jul 29 15:17:05.431749 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:05.431612    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b000ba3c-f29c-4314-877d-15246ef14307" containerName="maas-api"
Jul 29 15:17:05.431749 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:05.431623    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d84a00f9-6797-48be-9870-75a860426818" containerName="istio-proxy"
Jul 29 15:17:05.431749 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:05.431629    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d84a00f9-6797-48be-9870-75a860426818" containerName="istio-proxy"
Jul 29 15:17:05.431749 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:05.431646    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="042d3379-8772-400f-b264-b8597b7e0bc2" containerName="maas-api"
Jul 29 15:17:05.431749 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:05.431652    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="042d3379-8772-400f-b264-b8597b7e0bc2" containerName="maas-api"
Jul 29 15:17:05.431749 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:05.431661    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d84a00f9-6797-48be-9870-75a860426818" containerName="istio-proxy"
Jul 29 15:17:05.431749 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:05.431666    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d84a00f9-6797-48be-9870-75a860426818" containerName="istio-proxy"
Jul 29 15:17:05.431749 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:05.431728    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="042d3379-8772-400f-b264-b8597b7e0bc2" containerName="maas-api"
Jul 29 15:17:05.431749 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:05.431735    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b000ba3c-f29c-4314-877d-15246ef14307" containerName="maas-api"
Jul 29 15:17:05.431749 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:05.431741    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d84a00f9-6797-48be-9870-75a860426818" containerName="istio-proxy"
Jul 29 15:17:05.431749 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:05.431748    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d84a00f9-6797-48be-9870-75a860426818" containerName="istio-proxy"
Jul 29 15:17:05.436794 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:05.436773    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c73214-6b499bc688-pc4tt"
Jul 29 15:17:05.438815 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:05.438794    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-c73214-serving-cert\""
Jul 29 15:17:05.447373 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:05.447352    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c73214-6b499bc688-pc4tt"]
Jul 29 15:17:05.562400 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:05.562360    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39f22377-bf8f-40ed-a965-d38999419d95-maas-api-tls\") pod \"maas-api-e2e-shared-a-c73214-6b499bc688-pc4tt\" (UID: \"39f22377-bf8f-40ed-a965-d38999419d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c73214-6b499bc688-pc4tt"
Jul 29 15:17:05.562559 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:05.562408    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp79s\" (UniqueName: \"kubernetes.io/projected/39f22377-bf8f-40ed-a965-d38999419d95-kube-api-access-vp79s\") pod \"maas-api-e2e-shared-a-c73214-6b499bc688-pc4tt\" (UID: \"39f22377-bf8f-40ed-a965-d38999419d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c73214-6b499bc688-pc4tt"
Jul 29 15:17:05.562559 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:05.562429    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39f22377-bf8f-40ed-a965-d38999419d95-openshift-service-ca\") pod \"maas-api-e2e-shared-a-c73214-6b499bc688-pc4tt\" (UID: \"39f22377-bf8f-40ed-a965-d38999419d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c73214-6b499bc688-pc4tt"
Jul 29 15:17:05.663481 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:05.663438    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39f22377-bf8f-40ed-a965-d38999419d95-maas-api-tls\") pod \"maas-api-e2e-shared-a-c73214-6b499bc688-pc4tt\" (UID: \"39f22377-bf8f-40ed-a965-d38999419d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c73214-6b499bc688-pc4tt"
Jul 29 15:17:05.663631 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:05.663495    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp79s\" (UniqueName: \"kubernetes.io/projected/39f22377-bf8f-40ed-a965-d38999419d95-kube-api-access-vp79s\") pod \"maas-api-e2e-shared-a-c73214-6b499bc688-pc4tt\" (UID: \"39f22377-bf8f-40ed-a965-d38999419d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c73214-6b499bc688-pc4tt"
Jul 29 15:17:05.663631 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:05.663528    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39f22377-bf8f-40ed-a965-d38999419d95-openshift-service-ca\") pod \"maas-api-e2e-shared-a-c73214-6b499bc688-pc4tt\" (UID: \"39f22377-bf8f-40ed-a965-d38999419d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c73214-6b499bc688-pc4tt"
Jul 29 15:17:05.663631 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:17:05.663587    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-c73214-serving-cert: secret "maas-api-e2e-shared-a-c73214-serving-cert" not found
Jul 29 15:17:05.663756 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:17:05.663657    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39f22377-bf8f-40ed-a965-d38999419d95-maas-api-tls podName:39f22377-bf8f-40ed-a965-d38999419d95 nodeName:}" failed. No retries permitted until 2026-07-29 15:17:06.163636125 +0000 UTC m=+2528.212094041 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/39f22377-bf8f-40ed-a965-d38999419d95-maas-api-tls") pod "maas-api-e2e-shared-a-c73214-6b499bc688-pc4tt" (UID: "39f22377-bf8f-40ed-a965-d38999419d95") : secret "maas-api-e2e-shared-a-c73214-serving-cert" not found
Jul 29 15:17:05.664224 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:05.664206    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39f22377-bf8f-40ed-a965-d38999419d95-openshift-service-ca\") pod \"maas-api-e2e-shared-a-c73214-6b499bc688-pc4tt\" (UID: \"39f22377-bf8f-40ed-a965-d38999419d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c73214-6b499bc688-pc4tt"
Jul 29 15:17:05.672526 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:05.672503    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp79s\" (UniqueName: \"kubernetes.io/projected/39f22377-bf8f-40ed-a965-d38999419d95-kube-api-access-vp79s\") pod \"maas-api-e2e-shared-a-c73214-6b499bc688-pc4tt\" (UID: \"39f22377-bf8f-40ed-a965-d38999419d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c73214-6b499bc688-pc4tt"
Jul 29 15:17:06.168179 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:06.168124    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39f22377-bf8f-40ed-a965-d38999419d95-maas-api-tls\") pod \"maas-api-e2e-shared-a-c73214-6b499bc688-pc4tt\" (UID: \"39f22377-bf8f-40ed-a965-d38999419d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c73214-6b499bc688-pc4tt"
Jul 29 15:17:06.171381 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:06.171353    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39f22377-bf8f-40ed-a965-d38999419d95-maas-api-tls\") pod \"maas-api-e2e-shared-a-c73214-6b499bc688-pc4tt\" (UID: \"39f22377-bf8f-40ed-a965-d38999419d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c73214-6b499bc688-pc4tt"
Jul 29 15:17:06.348246 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:06.348202    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c73214-6b499bc688-pc4tt"
Jul 29 15:17:06.694428 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:06.694401    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c73214-6b499bc688-pc4tt"]
Jul 29 15:17:06.696941 ip-10-0-136-196 kubenswrapper[2563]: W0729 15:17:06.696916    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39f22377_bf8f_40ed_a965_d38999419d95.slice/crio-beba39f569abdaf9c149de8eb8b596edab1b73ab3d9a7939095535930a60b771 WatchSource:0}: Error finding container beba39f569abdaf9c149de8eb8b596edab1b73ab3d9a7939095535930a60b771: Status 404 returned error can't find the container with id beba39f569abdaf9c149de8eb8b596edab1b73ab3d9a7939095535930a60b771
Jul 29 15:17:07.189118 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.189095    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"
Jul 29 15:17:07.278694 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.278622    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330963f5-4a33-444c-bc0c-8832a9958c3f-workload-certs\") pod \"330963f5-4a33-444c-bc0c-8832a9958c3f\" (UID: \"330963f5-4a33-444c-bc0c-8832a9958c3f\") "
Jul 29 15:17:07.278694 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.278657    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330963f5-4a33-444c-bc0c-8832a9958c3f-istio-podinfo\") pod \"330963f5-4a33-444c-bc0c-8832a9958c3f\" (UID: \"330963f5-4a33-444c-bc0c-8832a9958c3f\") "
Jul 29 15:17:07.278694 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.278678    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330963f5-4a33-444c-bc0c-8832a9958c3f-istio-envoy\") pod \"330963f5-4a33-444c-bc0c-8832a9958c3f\" (UID: \"330963f5-4a33-444c-bc0c-8832a9958c3f\") "
Jul 29 15:17:07.278976 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.278710    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fs54d\" (UniqueName: \"kubernetes.io/projected/330963f5-4a33-444c-bc0c-8832a9958c3f-kube-api-access-fs54d\") pod \"330963f5-4a33-444c-bc0c-8832a9958c3f\" (UID: \"330963f5-4a33-444c-bc0c-8832a9958c3f\") "
Jul 29 15:17:07.278976 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.278728    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330963f5-4a33-444c-bc0c-8832a9958c3f-istiod-ca-cert\") pod \"330963f5-4a33-444c-bc0c-8832a9958c3f\" (UID: \"330963f5-4a33-444c-bc0c-8832a9958c3f\") "
Jul 29 15:17:07.278976 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.278855    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330963f5-4a33-444c-bc0c-8832a9958c3f-credential-socket\") pod \"330963f5-4a33-444c-bc0c-8832a9958c3f\" (UID: \"330963f5-4a33-444c-bc0c-8832a9958c3f\") "
Jul 29 15:17:07.278976 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.278911    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330963f5-4a33-444c-bc0c-8832a9958c3f-workload-socket\") pod \"330963f5-4a33-444c-bc0c-8832a9958c3f\" (UID: \"330963f5-4a33-444c-bc0c-8832a9958c3f\") "
Jul 29 15:17:07.278976 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.278926    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330963f5-4a33-444c-bc0c-8832a9958c3f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "330963f5-4a33-444c-bc0c-8832a9958c3f" (UID: "330963f5-4a33-444c-bc0c-8832a9958c3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:17:07.279404 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.279009    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330963f5-4a33-444c-bc0c-8832a9958c3f-istio-token\") pod \"330963f5-4a33-444c-bc0c-8832a9958c3f\" (UID: \"330963f5-4a33-444c-bc0c-8832a9958c3f\") "
Jul 29 15:17:07.279404 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.279044    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330963f5-4a33-444c-bc0c-8832a9958c3f-istio-data\") pod \"330963f5-4a33-444c-bc0c-8832a9958c3f\" (UID: \"330963f5-4a33-444c-bc0c-8832a9958c3f\") "
Jul 29 15:17:07.279404 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.279119    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330963f5-4a33-444c-bc0c-8832a9958c3f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "330963f5-4a33-444c-bc0c-8832a9958c3f" (UID: "330963f5-4a33-444c-bc0c-8832a9958c3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:17:07.279404 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.279150    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330963f5-4a33-444c-bc0c-8832a9958c3f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "330963f5-4a33-444c-bc0c-8832a9958c3f" (UID: "330963f5-4a33-444c-bc0c-8832a9958c3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:17:07.279404 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.279196    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/330963f5-4a33-444c-bc0c-8832a9958c3f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "330963f5-4a33-444c-bc0c-8832a9958c3f" (UID: "330963f5-4a33-444c-bc0c-8832a9958c3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 15:17:07.279667 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.279406    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330963f5-4a33-444c-bc0c-8832a9958c3f-credential-socket\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:17:07.279667 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.279426    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330963f5-4a33-444c-bc0c-8832a9958c3f-workload-socket\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:17:07.279667 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.279442    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330963f5-4a33-444c-bc0c-8832a9958c3f-workload-certs\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:17:07.279667 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.279455    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330963f5-4a33-444c-bc0c-8832a9958c3f-istiod-ca-cert\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:17:07.279916 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.279896    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330963f5-4a33-444c-bc0c-8832a9958c3f-istio-data" (OuterVolumeSpecName: "istio-data") pod "330963f5-4a33-444c-bc0c-8832a9958c3f" (UID: "330963f5-4a33-444c-bc0c-8832a9958c3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:17:07.281016 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.280995    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/330963f5-4a33-444c-bc0c-8832a9958c3f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "330963f5-4a33-444c-bc0c-8832a9958c3f" (UID: "330963f5-4a33-444c-bc0c-8832a9958c3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 15:17:07.281119 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.281072    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/330963f5-4a33-444c-bc0c-8832a9958c3f-kube-api-access-fs54d" (OuterVolumeSpecName: "kube-api-access-fs54d") pod "330963f5-4a33-444c-bc0c-8832a9958c3f" (UID: "330963f5-4a33-444c-bc0c-8832a9958c3f"). InnerVolumeSpecName "kube-api-access-fs54d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 15:17:07.281278 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.281257    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330963f5-4a33-444c-bc0c-8832a9958c3f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "330963f5-4a33-444c-bc0c-8832a9958c3f" (UID: "330963f5-4a33-444c-bc0c-8832a9958c3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:17:07.281318 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.281309    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/330963f5-4a33-444c-bc0c-8832a9958c3f-istio-token" (OuterVolumeSpecName: "istio-token") pod "330963f5-4a33-444c-bc0c-8832a9958c3f" (UID: "330963f5-4a33-444c-bc0c-8832a9958c3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 15:17:07.380102 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.380074    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330963f5-4a33-444c-bc0c-8832a9958c3f-istio-token\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:17:07.380102 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.380096    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330963f5-4a33-444c-bc0c-8832a9958c3f-istio-data\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:17:07.380102 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.380105    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330963f5-4a33-444c-bc0c-8832a9958c3f-istio-podinfo\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:17:07.380315 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.380113    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330963f5-4a33-444c-bc0c-8832a9958c3f-istio-envoy\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:17:07.380315 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.380123    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs54d\" (UniqueName: \"kubernetes.io/projected/330963f5-4a33-444c-bc0c-8832a9958c3f-kube-api-access-fs54d\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:17:07.435400 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.435365    2563 generic.go:356] "Generic (PLEG): container finished" podID="330963f5-4a33-444c-bc0c-8832a9958c3f" containerID="a9c03f15f5acebfed0ccf212a23185ca8cc6d850c5f602462037bf4aaac69f15" exitCode=0
Jul 29 15:17:07.435576 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.435420    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"
Jul 29 15:17:07.435576 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.435445    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk" event={"ID":"330963f5-4a33-444c-bc0c-8832a9958c3f","Type":"ContainerDied","Data":"a9c03f15f5acebfed0ccf212a23185ca8cc6d850c5f602462037bf4aaac69f15"}
Jul 29 15:17:07.435576 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.435501    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk" event={"ID":"330963f5-4a33-444c-bc0c-8832a9958c3f","Type":"ContainerDied","Data":"bb405ea55bc862ffc7f51c8295fac8d8b068c408801235f2cfd61861b0a7fd4e"}
Jul 29 15:17:07.435576 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.435523    2563 scope.go:117] "RemoveContainer" containerID="a9c03f15f5acebfed0ccf212a23185ca8cc6d850c5f602462037bf4aaac69f15"
Jul 29 15:17:07.437187 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.437120    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c73214-6b499bc688-pc4tt" event={"ID":"39f22377-bf8f-40ed-a965-d38999419d95","Type":"ContainerStarted","Data":"768951dc6cd5a0dc6902eda129d9d1a30b2a1d3202a83264903341f595632471"}
Jul 29 15:17:07.437299 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.437214    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c73214-6b499bc688-pc4tt"
Jul 29 15:17:07.437299 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.437240    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c73214-6b499bc688-pc4tt" event={"ID":"39f22377-bf8f-40ed-a965-d38999419d95","Type":"ContainerStarted","Data":"beba39f569abdaf9c149de8eb8b596edab1b73ab3d9a7939095535930a60b771"}
Jul 29 15:17:07.445511 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.445491    2563 scope.go:117] "RemoveContainer" containerID="feb0a563037f7556bb7a9750cf5afe76f4eca9339922265d5da38cc9e7b1af64"
Jul 29 15:17:07.453180 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.453161    2563 scope.go:117] "RemoveContainer" containerID="a9c03f15f5acebfed0ccf212a23185ca8cc6d850c5f602462037bf4aaac69f15"
Jul 29 15:17:07.453459 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:17:07.453439    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9c03f15f5acebfed0ccf212a23185ca8cc6d850c5f602462037bf4aaac69f15\": container with ID starting with a9c03f15f5acebfed0ccf212a23185ca8cc6d850c5f602462037bf4aaac69f15 not found: ID does not exist" containerID="a9c03f15f5acebfed0ccf212a23185ca8cc6d850c5f602462037bf4aaac69f15"
Jul 29 15:17:07.453556 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.453465    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9c03f15f5acebfed0ccf212a23185ca8cc6d850c5f602462037bf4aaac69f15"} err="failed to get container status \"a9c03f15f5acebfed0ccf212a23185ca8cc6d850c5f602462037bf4aaac69f15\": rpc error: code = NotFound desc = could not find container \"a9c03f15f5acebfed0ccf212a23185ca8cc6d850c5f602462037bf4aaac69f15\": container with ID starting with a9c03f15f5acebfed0ccf212a23185ca8cc6d850c5f602462037bf4aaac69f15 not found: ID does not exist"
Jul 29 15:17:07.453556 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.453483    2563 scope.go:117] "RemoveContainer" containerID="feb0a563037f7556bb7a9750cf5afe76f4eca9339922265d5da38cc9e7b1af64"
Jul 29 15:17:07.453921 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:17:07.453842    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feb0a563037f7556bb7a9750cf5afe76f4eca9339922265d5da38cc9e7b1af64\": container with ID starting with feb0a563037f7556bb7a9750cf5afe76f4eca9339922265d5da38cc9e7b1af64 not found: ID does not exist" containerID="feb0a563037f7556bb7a9750cf5afe76f4eca9339922265d5da38cc9e7b1af64"
Jul 29 15:17:07.453921 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.453874    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feb0a563037f7556bb7a9750cf5afe76f4eca9339922265d5da38cc9e7b1af64"} err="failed to get container status \"feb0a563037f7556bb7a9750cf5afe76f4eca9339922265d5da38cc9e7b1af64\": rpc error: code = NotFound desc = could not find container \"feb0a563037f7556bb7a9750cf5afe76f4eca9339922265d5da38cc9e7b1af64\": container with ID starting with feb0a563037f7556bb7a9750cf5afe76f4eca9339922265d5da38cc9e7b1af64 not found: ID does not exist"
Jul 29 15:17:07.454730 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.454686    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c73214-6b499bc688-pc4tt" podStartSLOduration=2.041755968 podStartE2EDuration="2.45466955s" totalImagesPullingTime="412.913582ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:17:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:17:07.45354375 +0000 UTC m=+2529.502001690" watchObservedRunningTime="2026-07-29 15:17:07.45466955 +0000 UTC m=+2529.503127491"
Jul 29 15:17:07.469305 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.469281    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"]
Jul 29 15:17:07.473881 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:07.473858    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d3e823-openshift-default-5b57879b76-twlfk"]
Jul 29 15:17:08.432206 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:08.432174    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="330963f5-4a33-444c-bc0c-8832a9958c3f" path="/var/lib/kubelet/pods/330963f5-4a33-444c-bc0c-8832a9958c3f/volumes"
Jul 29 15:17:13.447663 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:13.447628    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c73214-6b499bc688-pc4tt"
Jul 29 15:17:31.092594 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.092560    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"]
Jul 29 15:17:31.093179 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.092942    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="330963f5-4a33-444c-bc0c-8832a9958c3f" containerName="istio-proxy"
Jul 29 15:17:31.093179 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.092955    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="330963f5-4a33-444c-bc0c-8832a9958c3f" containerName="istio-proxy"
Jul 29 15:17:31.093179 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.092965    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="330963f5-4a33-444c-bc0c-8832a9958c3f" containerName="istio-proxy"
Jul 29 15:17:31.093179 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.092987    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="330963f5-4a33-444c-bc0c-8832a9958c3f" containerName="istio-proxy"
Jul 29 15:17:31.093179 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.093058    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="330963f5-4a33-444c-bc0c-8832a9958c3f" containerName="istio-proxy"
Jul 29 15:17:31.097493 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.097475    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"
Jul 29 15:17:31.100120 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.100095    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 29 15:17:31.100275 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.100158    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-93e799-openshift-default-dockercfg-f28bk\""
Jul 29 15:17:31.108186 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.108159    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"]
Jul 29 15:17:31.170605 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.170565    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a406c1c9-fcaf-4d16-aa78-0383a59e3260-credential-socket\") pod \"e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7\" (UID: \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\") " pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"
Jul 29 15:17:31.170777 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.170610    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a406c1c9-fcaf-4d16-aa78-0383a59e3260-workload-socket\") pod \"e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7\" (UID: \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\") " pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"
Jul 29 15:17:31.170777 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.170635    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg4r8\" (UniqueName: \"kubernetes.io/projected/a406c1c9-fcaf-4d16-aa78-0383a59e3260-kube-api-access-vg4r8\") pod \"e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7\" (UID: \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\") " pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"
Jul 29 15:17:31.170777 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.170700    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a406c1c9-fcaf-4d16-aa78-0383a59e3260-istio-token\") pod \"e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7\" (UID: \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\") " pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"
Jul 29 15:17:31.170777 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.170727    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a406c1c9-fcaf-4d16-aa78-0383a59e3260-istio-envoy\") pod \"e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7\" (UID: \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\") " pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"
Jul 29 15:17:31.170777 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.170743    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a406c1c9-fcaf-4d16-aa78-0383a59e3260-istio-data\") pod \"e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7\" (UID: \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\") " pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"
Jul 29 15:17:31.170777 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.170769    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a406c1c9-fcaf-4d16-aa78-0383a59e3260-istio-podinfo\") pod \"e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7\" (UID: \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\") " pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"
Jul 29 15:17:31.170992 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.170792    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a406c1c9-fcaf-4d16-aa78-0383a59e3260-istiod-ca-cert\") pod \"e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7\" (UID: \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\") " pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"
Jul 29 15:17:31.170992 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.170815    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a406c1c9-fcaf-4d16-aa78-0383a59e3260-workload-certs\") pod \"e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7\" (UID: \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\") " pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"
Jul 29 15:17:31.271383 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.271345    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a406c1c9-fcaf-4d16-aa78-0383a59e3260-istio-token\") pod \"e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7\" (UID: \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\") " pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"
Jul 29 15:17:31.271383 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.271388    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a406c1c9-fcaf-4d16-aa78-0383a59e3260-istio-envoy\") pod \"e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7\" (UID: \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\") " pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"
Jul 29 15:17:31.271650 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.271414    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a406c1c9-fcaf-4d16-aa78-0383a59e3260-istio-data\") pod \"e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7\" (UID: \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\") " pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"
Jul 29 15:17:31.271650 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.271442    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a406c1c9-fcaf-4d16-aa78-0383a59e3260-istio-podinfo\") pod \"e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7\" (UID: \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\") " pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"
Jul 29 15:17:31.271650 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.271462    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a406c1c9-fcaf-4d16-aa78-0383a59e3260-istiod-ca-cert\") pod \"e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7\" (UID: \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\") " pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"
Jul 29 15:17:31.271650 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.271489    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a406c1c9-fcaf-4d16-aa78-0383a59e3260-workload-certs\") pod \"e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7\" (UID: \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\") " pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"
Jul 29 15:17:31.271915 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.271891    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a406c1c9-fcaf-4d16-aa78-0383a59e3260-istio-data\") pod \"e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7\" (UID: \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\") " pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"
Jul 29 15:17:31.271979 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.271938    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a406c1c9-fcaf-4d16-aa78-0383a59e3260-workload-certs\") pod \"e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7\" (UID: \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\") " pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"
Jul 29 15:17:31.272179 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.272158    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a406c1c9-fcaf-4d16-aa78-0383a59e3260-credential-socket\") pod \"e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7\" (UID: \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\") " pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"
Jul 29 15:17:31.272261 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.272197    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a406c1c9-fcaf-4d16-aa78-0383a59e3260-workload-socket\") pod \"e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7\" (UID: \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\") " pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"
Jul 29 15:17:31.272261 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.272229    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg4r8\" (UniqueName: \"kubernetes.io/projected/a406c1c9-fcaf-4d16-aa78-0383a59e3260-kube-api-access-vg4r8\") pod \"e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7\" (UID: \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\") " pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"
Jul 29 15:17:31.272422 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.272252    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a406c1c9-fcaf-4d16-aa78-0383a59e3260-istiod-ca-cert\") pod \"e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7\" (UID: \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\") " pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"
Jul 29 15:17:31.272511 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.272492    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a406c1c9-fcaf-4d16-aa78-0383a59e3260-credential-socket\") pod \"e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7\" (UID: \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\") " pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"
Jul 29 15:17:31.272571 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.272516    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a406c1c9-fcaf-4d16-aa78-0383a59e3260-workload-socket\") pod \"e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7\" (UID: \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\") " pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"
Jul 29 15:17:31.274297 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.274271    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a406c1c9-fcaf-4d16-aa78-0383a59e3260-istio-envoy\") pod \"e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7\" (UID: \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\") " pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"
Jul 29 15:17:31.274422 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.274406    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a406c1c9-fcaf-4d16-aa78-0383a59e3260-istio-podinfo\") pod \"e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7\" (UID: \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\") " pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"
Jul 29 15:17:31.279682 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.279659    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a406c1c9-fcaf-4d16-aa78-0383a59e3260-istio-token\") pod \"e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7\" (UID: \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\") " pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"
Jul 29 15:17:31.280213 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.280197    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg4r8\" (UniqueName: \"kubernetes.io/projected/a406c1c9-fcaf-4d16-aa78-0383a59e3260-kube-api-access-vg4r8\") pod \"e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7\" (UID: \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\") " pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"
Jul 29 15:17:31.408864 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.408758    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"
Jul 29 15:17:31.551840 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.551809    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"]
Jul 29 15:17:31.553319 ip-10-0-136-196 kubenswrapper[2563]: W0729 15:17:31.553291    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda406c1c9_fcaf_4d16_aa78_0383a59e3260.slice/crio-08405a7cd44d077820bc59d11e9463f3ad7ea943d9e210748ce8a29aba73181c WatchSource:0}: Error finding container 08405a7cd44d077820bc59d11e9463f3ad7ea943d9e210748ce8a29aba73181c: Status 404 returned error can't find the container with id 08405a7cd44d077820bc59d11e9463f3ad7ea943d9e210748ce8a29aba73181c
Jul 29 15:17:31.555351 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.555320    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:17:31.555443 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.555382    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:17:31.555443 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:31.555412    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:17:32.554489 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:32.554449    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7" event={"ID":"a406c1c9-fcaf-4d16-aa78-0383a59e3260","Type":"ContainerStarted","Data":"c94826643f6a1bcc1b1e86cfd07a82188eb216571fc6a0769efe37e0afba8292"}
Jul 29 15:17:32.554960 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:32.554546    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7" event={"ID":"a406c1c9-fcaf-4d16-aa78-0383a59e3260","Type":"ContainerStarted","Data":"08405a7cd44d077820bc59d11e9463f3ad7ea943d9e210748ce8a29aba73181c"}
Jul 29 15:17:32.578970 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:32.578912    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7" podStartSLOduration=1.5788961609999999 podStartE2EDuration="1.578896161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:17:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:17:32.577678676 +0000 UTC m=+2554.626136615" watchObservedRunningTime="2026-07-29 15:17:32.578896161 +0000 UTC m=+2554.627354099"
Jul 29 15:17:33.409345 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:33.409299    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"
Jul 29 15:17:33.414595 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:33.414563    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"
Jul 29 15:17:33.559103 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:33.559075    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"
Jul 29 15:17:33.560073 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:33.560051    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"
Jul 29 15:17:34.684077 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:34.684039    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-93e799-5bdd9c84bc-r8l9m"]
Jul 29 15:17:34.684530 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:34.684516    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="330963f5-4a33-444c-bc0c-8832a9958c3f" containerName="istio-proxy"
Jul 29 15:17:34.687675 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:34.687647    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-93e799-5bdd9c84bc-r8l9m"
Jul 29 15:17:34.690110 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:34.690092    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-93e799-serving-cert\""
Jul 29 15:17:34.706610 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:34.706578    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-93e799-5bdd9c84bc-r8l9m"]
Jul 29 15:17:34.805409 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:34.805372    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddlxh\" (UniqueName: \"kubernetes.io/projected/b09574c0-d7c9-4cef-99f9-ca1cadab0af6-kube-api-access-ddlxh\") pod \"maas-api-e2e-shared-b-93e799-5bdd9c84bc-r8l9m\" (UID: \"b09574c0-d7c9-4cef-99f9-ca1cadab0af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-93e799-5bdd9c84bc-r8l9m"
Jul 29 15:17:34.805605 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:34.805424    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b09574c0-d7c9-4cef-99f9-ca1cadab0af6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-93e799-5bdd9c84bc-r8l9m\" (UID: \"b09574c0-d7c9-4cef-99f9-ca1cadab0af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-93e799-5bdd9c84bc-r8l9m"
Jul 29 15:17:34.805605 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:34.805484    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b09574c0-d7c9-4cef-99f9-ca1cadab0af6-maas-api-tls\") pod \"maas-api-e2e-shared-b-93e799-5bdd9c84bc-r8l9m\" (UID: \"b09574c0-d7c9-4cef-99f9-ca1cadab0af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-93e799-5bdd9c84bc-r8l9m"
Jul 29 15:17:34.906810 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:34.906778    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddlxh\" (UniqueName: \"kubernetes.io/projected/b09574c0-d7c9-4cef-99f9-ca1cadab0af6-kube-api-access-ddlxh\") pod \"maas-api-e2e-shared-b-93e799-5bdd9c84bc-r8l9m\" (UID: \"b09574c0-d7c9-4cef-99f9-ca1cadab0af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-93e799-5bdd9c84bc-r8l9m"
Jul 29 15:17:34.907000 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:34.906824    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b09574c0-d7c9-4cef-99f9-ca1cadab0af6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-93e799-5bdd9c84bc-r8l9m\" (UID: \"b09574c0-d7c9-4cef-99f9-ca1cadab0af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-93e799-5bdd9c84bc-r8l9m"
Jul 29 15:17:34.907000 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:34.906858    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b09574c0-d7c9-4cef-99f9-ca1cadab0af6-maas-api-tls\") pod \"maas-api-e2e-shared-b-93e799-5bdd9c84bc-r8l9m\" (UID: \"b09574c0-d7c9-4cef-99f9-ca1cadab0af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-93e799-5bdd9c84bc-r8l9m"
Jul 29 15:17:34.907559 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:34.907536    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b09574c0-d7c9-4cef-99f9-ca1cadab0af6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-93e799-5bdd9c84bc-r8l9m\" (UID: \"b09574c0-d7c9-4cef-99f9-ca1cadab0af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-93e799-5bdd9c84bc-r8l9m"
Jul 29 15:17:34.909331 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:34.909313    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b09574c0-d7c9-4cef-99f9-ca1cadab0af6-maas-api-tls\") pod \"maas-api-e2e-shared-b-93e799-5bdd9c84bc-r8l9m\" (UID: \"b09574c0-d7c9-4cef-99f9-ca1cadab0af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-93e799-5bdd9c84bc-r8l9m"
Jul 29 15:17:34.915077 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:34.915053    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddlxh\" (UniqueName: \"kubernetes.io/projected/b09574c0-d7c9-4cef-99f9-ca1cadab0af6-kube-api-access-ddlxh\") pod \"maas-api-e2e-shared-b-93e799-5bdd9c84bc-r8l9m\" (UID: \"b09574c0-d7c9-4cef-99f9-ca1cadab0af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-93e799-5bdd9c84bc-r8l9m"
Jul 29 15:17:34.998971 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:34.998937    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-93e799-5bdd9c84bc-r8l9m"
Jul 29 15:17:35.142675 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:35.142644    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-93e799-5bdd9c84bc-r8l9m"]
Jul 29 15:17:35.145946 ip-10-0-136-196 kubenswrapper[2563]: W0729 15:17:35.145917    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb09574c0_d7c9_4cef_99f9_ca1cadab0af6.slice/crio-62b12eae6b77f141cc004ed2bff231cac468024008f63f06ffc957ef45acf7e6 WatchSource:0}: Error finding container 62b12eae6b77f141cc004ed2bff231cac468024008f63f06ffc957ef45acf7e6: Status 404 returned error can't find the container with id 62b12eae6b77f141cc004ed2bff231cac468024008f63f06ffc957ef45acf7e6
Jul 29 15:17:35.570519 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:35.570480    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-93e799-5bdd9c84bc-r8l9m" event={"ID":"b09574c0-d7c9-4cef-99f9-ca1cadab0af6","Type":"ContainerStarted","Data":"23a5214f7551f5ae6afaa89073fb842ff21e4d24a4cdfccb7673dae51bb18e71"}
Jul 29 15:17:35.570726 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:35.570569    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-93e799-5bdd9c84bc-r8l9m" event={"ID":"b09574c0-d7c9-4cef-99f9-ca1cadab0af6","Type":"ContainerStarted","Data":"62b12eae6b77f141cc004ed2bff231cac468024008f63f06ffc957ef45acf7e6"}
Jul 29 15:17:35.570814 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:35.570778    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-93e799-5bdd9c84bc-r8l9m"
Jul 29 15:17:35.589609 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:35.589557    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-93e799-5bdd9c84bc-r8l9m" podStartSLOduration=1.264352664 podStartE2EDuration="1.589542562s" totalImagesPullingTime="325.189898ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:17:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:17:35.587026912 +0000 UTC m=+2557.635484852" watchObservedRunningTime="2026-07-29 15:17:35.589542562 +0000 UTC m=+2557.638000501"
Jul 29 15:17:41.580302 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:41.580272    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-93e799-5bdd9c84bc-r8l9m"
Jul 29 15:17:49.601303 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:49.601258    2563 patch_prober.go:28] interesting pod/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 29 15:17:49.601710 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:49.601319    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7" podUID="a406c1c9-fcaf-4d16-aa78-0383a59e3260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 29 15:17:50.649703 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:50.649671    2563 generic.go:356] "Generic (PLEG): container finished" podID="a406c1c9-fcaf-4d16-aa78-0383a59e3260" containerID="c94826643f6a1bcc1b1e86cfd07a82188eb216571fc6a0769efe37e0afba8292" exitCode=0
Jul 29 15:17:50.650128 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:50.649746    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7" event={"ID":"a406c1c9-fcaf-4d16-aa78-0383a59e3260","Type":"ContainerDied","Data":"c94826643f6a1bcc1b1e86cfd07a82188eb216571fc6a0769efe37e0afba8292"}
Jul 29 15:17:50.650214 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:50.650162    2563 scope.go:117] "RemoveContainer" containerID="c94826643f6a1bcc1b1e86cfd07a82188eb216571fc6a0769efe37e0afba8292"
Jul 29 15:17:50.651189 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:50.651162    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:17:50.651255 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:50.651228    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:17:50.651297 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:50.651262    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:17:51.661887 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:51.661845    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7" event={"ID":"a406c1c9-fcaf-4d16-aa78-0383a59e3260","Type":"ContainerStarted","Data":"dfec5e727ddb4d2b77e7c2fac4fa0fdbc7430a0c5e3446f033e2d21a01588182"}
Jul 29 15:17:52.409832 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:52.409789    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"
Jul 29 15:17:52.410020 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:52.410000    2563 patch_prober.go:28] interesting pod/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Jul 29 15:17:52.410074 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:52.410052    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7" podUID="a406c1c9-fcaf-4d16-aa78-0383a59e3260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Jul 29 15:17:53.410220 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:53.410178    2563 patch_prober.go:28] interesting pod/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Jul 29 15:17:53.410625 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:53.410272    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7" podUID="a406c1c9-fcaf-4d16-aa78-0383a59e3260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Jul 29 15:17:54.409575 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:54.409538    2563 patch_prober.go:28] interesting pod/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Jul 29 15:17:54.409756 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:54.409593    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7" podUID="a406c1c9-fcaf-4d16-aa78-0383a59e3260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Jul 29 15:17:55.409495 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:55.409459    2563 patch_prober.go:28] interesting pod/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Jul 29 15:17:55.409882 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:55.409517    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7" podUID="a406c1c9-fcaf-4d16-aa78-0383a59e3260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Jul 29 15:17:56.413199 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:56.413168    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"
Jul 29 15:17:56.413584 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:56.413450    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"
Jul 29 15:17:56.414195 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:17:56.414177    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"
Jul 29 15:19:58.873147 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:19:58.873052    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xnm95_23d79588-937d-4c6d-8760-a081024f608e/console-operator/1.log"
Jul 29 15:19:58.874938 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:19:58.874910    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xnm95_23d79588-937d-4c6d-8760-a081024f608e/console-operator/1.log"
Jul 29 15:24:58.903102 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:24:58.903069    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xnm95_23d79588-937d-4c6d-8760-a081024f608e/console-operator/1.log"
Jul 29 15:24:58.905246 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:24:58.905215    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xnm95_23d79588-937d-4c6d-8760-a081024f608e/console-operator/1.log"
Jul 29 15:27:07.463770 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:07.463715    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7695-140291047943168"]
Jul 29 15:27:07.471055 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:07.471023    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7695-140291047943168"]
Jul 29 15:27:07.471247 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:07.471177    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-140291047943168"
Jul 29 15:27:07.473683 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:07.473659    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-n6sn7\""
Jul 29 15:27:07.552295 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:07.552254    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4672\" (UniqueName: \"kubernetes.io/projected/fb4530fd-975b-47ac-ba29-54f720c2be18-kube-api-access-w4672\") pod \"test-curl-7695-140291047943168\" (UID: \"fb4530fd-975b-47ac-ba29-54f720c2be18\") " pod="opendatahub/test-curl-7695-140291047943168"
Jul 29 15:27:07.653187 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:07.653125    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4672\" (UniqueName: \"kubernetes.io/projected/fb4530fd-975b-47ac-ba29-54f720c2be18-kube-api-access-w4672\") pod \"test-curl-7695-140291047943168\" (UID: \"fb4530fd-975b-47ac-ba29-54f720c2be18\") " pod="opendatahub/test-curl-7695-140291047943168"
Jul 29 15:27:07.660449 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:07.660415    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4672\" (UniqueName: \"kubernetes.io/projected/fb4530fd-975b-47ac-ba29-54f720c2be18-kube-api-access-w4672\") pod \"test-curl-7695-140291047943168\" (UID: \"fb4530fd-975b-47ac-ba29-54f720c2be18\") " pod="opendatahub/test-curl-7695-140291047943168"
Jul 29 15:27:07.783600 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:07.783510    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-140291047943168"
Jul 29 15:27:07.928694 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:07.928657    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7695-140291047943168"]
Jul 29 15:27:07.932730 ip-10-0-136-196 kubenswrapper[2563]: W0729 15:27:07.932702    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb4530fd_975b_47ac_ba29_54f720c2be18.slice/crio-edc32aef036f1c31b0351ebc505e906dde904da67e9694a7985fd5f93ba97db2 WatchSource:0}: Error finding container edc32aef036f1c31b0351ebc505e906dde904da67e9694a7985fd5f93ba97db2: Status 404 returned error can't find the container with id edc32aef036f1c31b0351ebc505e906dde904da67e9694a7985fd5f93ba97db2
Jul 29 15:27:07.934186 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:07.934164    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 15:27:08.408455 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:08.408413    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7695-140291047943168" event={"ID":"fb4530fd-975b-47ac-ba29-54f720c2be18","Type":"ContainerStarted","Data":"edc32aef036f1c31b0351ebc505e906dde904da67e9694a7985fd5f93ba97db2"}
Jul 29 15:27:09.414960 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:09.414919    2563 generic.go:356] "Generic (PLEG): container finished" podID="fb4530fd-975b-47ac-ba29-54f720c2be18" containerID="8072237f1e3769a42653df56c310a704fbee017830f92c073b313b50f5f24d0c" exitCode=0
Jul 29 15:27:09.415382 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:09.414992    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7695-140291047943168" event={"ID":"fb4530fd-975b-47ac-ba29-54f720c2be18","Type":"ContainerDied","Data":"8072237f1e3769a42653df56c310a704fbee017830f92c073b313b50f5f24d0c"}
Jul 29 15:27:10.560325 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:10.560296    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-140291047943168"
Jul 29 15:27:10.594128 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:10.594095    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7695-140291047943168_fb4530fd-975b-47ac-ba29-54f720c2be18/test-curl-7695-140291047943168/0.log"
Jul 29 15:27:10.617494 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:10.617453    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7695-140291047943168"]
Jul 29 15:27:10.621324 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:10.621289    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7695-140291047943168"]
Jul 29 15:27:10.683836 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:10.683733    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4672\" (UniqueName: \"kubernetes.io/projected/fb4530fd-975b-47ac-ba29-54f720c2be18-kube-api-access-w4672\") pod \"fb4530fd-975b-47ac-ba29-54f720c2be18\" (UID: \"fb4530fd-975b-47ac-ba29-54f720c2be18\") "
Jul 29 15:27:10.686042 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:10.686004    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb4530fd-975b-47ac-ba29-54f720c2be18-kube-api-access-w4672" (OuterVolumeSpecName: "kube-api-access-w4672") pod "fb4530fd-975b-47ac-ba29-54f720c2be18" (UID: "fb4530fd-975b-47ac-ba29-54f720c2be18"). InnerVolumeSpecName "kube-api-access-w4672". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 15:27:10.784752 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:10.784709    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4672\" (UniqueName: \"kubernetes.io/projected/fb4530fd-975b-47ac-ba29-54f720c2be18-kube-api-access-w4672\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:27:11.426164 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:11.426117    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-140291047943168"
Jul 29 15:27:11.426343 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:11.426126    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edc32aef036f1c31b0351ebc505e906dde904da67e9694a7985fd5f93ba97db2"
Jul 29 15:27:11.436222 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:27:11.436187    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7695-140291047943168\" is forbidden: User \"system:node:ip-10-0-136-196.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-196.ec2.internal' and this object" podUID="fb4530fd-975b-47ac-ba29-54f720c2be18" pod="opendatahub/test-curl-7695-140291047943168"
Jul 29 15:27:12.432650 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:12.432617    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb4530fd-975b-47ac-ba29-54f720c2be18" path="/var/lib/kubelet/pods/fb4530fd-975b-47ac-ba29-54f720c2be18/volumes"
Jul 29 15:27:14.189384 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:14.189342    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"]
Jul 29 15:27:14.189944 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:14.189926    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb4530fd-975b-47ac-ba29-54f720c2be18" containerName="test-curl-7695-140291047943168"
Jul 29 15:27:14.189944 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:14.189945    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb4530fd-975b-47ac-ba29-54f720c2be18" containerName="test-curl-7695-140291047943168"
Jul 29 15:27:14.190066 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:14.190013    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb4530fd-975b-47ac-ba29-54f720c2be18" containerName="test-curl-7695-140291047943168"
Jul 29 15:27:14.193286 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:14.193267    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"
Jul 29 15:27:14.195460 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:14.195435    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-3261e1-openshift-default-dockercfg-52wqm\""
Jul 29 15:27:14.205392 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:14.205363    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"]
Jul 29 15:27:14.316286 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:14.316239    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6746c008-6c97-4473-909b-c5c9e4c90b7a-workload-socket\") pod \"e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z\" (UID: \"6746c008-6c97-4473-909b-c5c9e4c90b7a\") " pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"
Jul 29 15:27:14.316486 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:14.316303    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6746c008-6c97-4473-909b-c5c9e4c90b7a-istio-podinfo\") pod \"e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z\" (UID: \"6746c008-6c97-4473-909b-c5c9e4c90b7a\") " pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"
Jul 29 15:27:14.316486 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:14.316322    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6746c008-6c97-4473-909b-c5c9e4c90b7a-istiod-ca-cert\") pod \"e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z\" (UID: \"6746c008-6c97-4473-909b-c5c9e4c90b7a\") " pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"
Jul 29 15:27:14.316486 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:14.316381    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6746c008-6c97-4473-909b-c5c9e4c90b7a-istio-token\") pod \"e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z\" (UID: \"6746c008-6c97-4473-909b-c5c9e4c90b7a\") " pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"
Jul 29 15:27:14.316486 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:14.316466    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6746c008-6c97-4473-909b-c5c9e4c90b7a-credential-socket\") pod \"e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z\" (UID: \"6746c008-6c97-4473-909b-c5c9e4c90b7a\") " pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"
Jul 29 15:27:14.316486 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:14.316487    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6746c008-6c97-4473-909b-c5c9e4c90b7a-istio-envoy\") pod \"e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z\" (UID: \"6746c008-6c97-4473-909b-c5c9e4c90b7a\") " pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"
Jul 29 15:27:14.316697 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:14.316503    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzdj9\" (UniqueName: \"kubernetes.io/projected/6746c008-6c97-4473-909b-c5c9e4c90b7a-kube-api-access-tzdj9\") pod \"e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z\" (UID: \"6746c008-6c97-4473-909b-c5c9e4c90b7a\") " pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"
Jul 29 15:27:14.316697 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:14.316522    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6746c008-6c97-4473-909b-c5c9e4c90b7a-workload-certs\") pod \"e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z\" (UID: \"6746c008-6c97-4473-909b-c5c9e4c90b7a\") " pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"
Jul 29 15:27:14.316697 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:14.316537    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6746c008-6c97-4473-909b-c5c9e4c90b7a-istio-data\") pod \"e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z\" (UID: \"6746c008-6c97-4473-909b-c5c9e4c90b7a\") " pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"
Jul 29 15:27:14.417224 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:14.417176    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6746c008-6c97-4473-909b-c5c9e4c90b7a-credential-socket\") pod \"e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z\" (UID: \"6746c008-6c97-4473-909b-c5c9e4c90b7a\") " pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"
Jul 29 15:27:14.417432 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:14.417235    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6746c008-6c97-4473-909b-c5c9e4c90b7a-istio-envoy\") pod \"e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z\" (UID: \"6746c008-6c97-4473-909b-c5c9e4c90b7a\") " pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"
Jul 29 15:27:14.417432 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:14.417259    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzdj9\" (UniqueName: \"kubernetes.io/projected/6746c008-6c97-4473-909b-c5c9e4c90b7a-kube-api-access-tzdj9\") pod \"e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z\" (UID: \"6746c008-6c97-4473-909b-c5c9e4c90b7a\") " pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"
Jul 29 15:27:14.417432 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:14.417280    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6746c008-6c97-4473-909b-c5c9e4c90b7a-workload-certs\") pod \"e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z\" (UID: \"6746c008-6c97-4473-909b-c5c9e4c90b7a\") " pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"
Jul 29 15:27:14.417432 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:14.417300    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6746c008-6c97-4473-909b-c5c9e4c90b7a-istio-data\") pod \"e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z\" (UID: \"6746c008-6c97-4473-909b-c5c9e4c90b7a\") " pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"
Jul 29 15:27:14.417432 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:14.417330    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6746c008-6c97-4473-909b-c5c9e4c90b7a-workload-socket\") pod \"e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z\" (UID: \"6746c008-6c97-4473-909b-c5c9e4c90b7a\") " pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"
Jul 29 15:27:14.417432 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:14.417389    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6746c008-6c97-4473-909b-c5c9e4c90b7a-istio-podinfo\") pod \"e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z\" (UID: \"6746c008-6c97-4473-909b-c5c9e4c90b7a\") " pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"
Jul 29 15:27:14.417432 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:14.417421    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6746c008-6c97-4473-909b-c5c9e4c90b7a-istiod-ca-cert\") pod \"e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z\" (UID: \"6746c008-6c97-4473-909b-c5c9e4c90b7a\") " pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"
Jul 29 15:27:14.417808 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:14.417464    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6746c008-6c97-4473-909b-c5c9e4c90b7a-istio-token\") pod \"e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z\" (UID: \"6746c008-6c97-4473-909b-c5c9e4c90b7a\") " pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"
Jul 29 15:27:14.417808 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:14.417606    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6746c008-6c97-4473-909b-c5c9e4c90b7a-credential-socket\") pod \"e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z\" (UID: \"6746c008-6c97-4473-909b-c5c9e4c90b7a\") " pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"
Jul 29 15:27:14.417808 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:14.417643    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6746c008-6c97-4473-909b-c5c9e4c90b7a-istio-data\") pod \"e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z\" (UID: \"6746c008-6c97-4473-909b-c5c9e4c90b7a\") " pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"
Jul 29 15:27:14.417974 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:14.417738    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6746c008-6c97-4473-909b-c5c9e4c90b7a-workload-certs\") pod \"e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z\" (UID: \"6746c008-6c97-4473-909b-c5c9e4c90b7a\") " pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"
Jul 29 15:27:14.417974 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:14.417923    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6746c008-6c97-4473-909b-c5c9e4c90b7a-workload-socket\") pod \"e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z\" (UID: \"6746c008-6c97-4473-909b-c5c9e4c90b7a\") " pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"
Jul 29 15:27:14.418497 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:14.418473    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6746c008-6c97-4473-909b-c5c9e4c90b7a-istiod-ca-cert\") pod \"e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z\" (UID: \"6746c008-6c97-4473-909b-c5c9e4c90b7a\") " pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"
Jul 29 15:27:14.419776 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:14.419743    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6746c008-6c97-4473-909b-c5c9e4c90b7a-istio-envoy\") pod \"e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z\" (UID: \"6746c008-6c97-4473-909b-c5c9e4c90b7a\") " pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"
Jul 29 15:27:14.420116 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:14.420093    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6746c008-6c97-4473-909b-c5c9e4c90b7a-istio-podinfo\") pod \"e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z\" (UID: \"6746c008-6c97-4473-909b-c5c9e4c90b7a\") " pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"
Jul 29 15:27:14.425125 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:14.425097    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6746c008-6c97-4473-909b-c5c9e4c90b7a-istio-token\") pod \"e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z\" (UID: \"6746c008-6c97-4473-909b-c5c9e4c90b7a\") " pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"
Jul 29 15:27:14.425458 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:14.425434    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzdj9\" (UniqueName: \"kubernetes.io/projected/6746c008-6c97-4473-909b-c5c9e4c90b7a-kube-api-access-tzdj9\") pod \"e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z\" (UID: \"6746c008-6c97-4473-909b-c5c9e4c90b7a\") " pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"
Jul 29 15:27:14.504904 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:14.504864    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"
Jul 29 15:27:14.657055 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:14.657012    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"]
Jul 29 15:27:14.661317 ip-10-0-136-196 kubenswrapper[2563]: W0729 15:27:14.661281    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6746c008_6c97_4473_909b_c5c9e4c90b7a.slice/crio-e1c0440a008670acf59de2f7125e5c845a937ac106c3c5f98eb55c0255082ff3 WatchSource:0}: Error finding container e1c0440a008670acf59de2f7125e5c845a937ac106c3c5f98eb55c0255082ff3: Status 404 returned error can't find the container with id e1c0440a008670acf59de2f7125e5c845a937ac106c3c5f98eb55c0255082ff3
Jul 29 15:27:14.663551 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:14.663514    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:27:14.663642 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:14.663585    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:27:14.663642 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:14.663616    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:27:15.449383 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:15.449341    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z" event={"ID":"6746c008-6c97-4473-909b-c5c9e4c90b7a","Type":"ContainerStarted","Data":"a1ed64ca1ab6d14da167d8a6adf975af1cb3ccda33685c158d2384595e3e5a05"}
Jul 29 15:27:15.449907 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:15.449419    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z" event={"ID":"6746c008-6c97-4473-909b-c5c9e4c90b7a","Type":"ContainerStarted","Data":"e1c0440a008670acf59de2f7125e5c845a937ac106c3c5f98eb55c0255082ff3"}
Jul 29 15:27:15.475971 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:15.475895    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z" podStartSLOduration=1.475872358 podStartE2EDuration="1.475872358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:27:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:27:15.472978526 +0000 UTC m=+3137.521436465" watchObservedRunningTime="2026-07-29 15:27:15.475872358 +0000 UTC m=+3137.524330298"
Jul 29 15:27:15.505772 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:15.505735    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"
Jul 29 15:27:15.510925 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:15.510892    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"
Jul 29 15:27:16.454371 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:16.454335    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"
Jul 29 15:27:16.455338 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:16.455316    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"
Jul 29 15:27:17.522931 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:17.522891    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3261e1-75b769cd58-wcvz5"]
Jul 29 15:27:17.527127 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:17.527098    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3261e1-75b769cd58-wcvz5"
Jul 29 15:27:17.529553 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:17.529453    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-3261e1-serving-cert\""
Jul 29 15:27:17.537368 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:17.537252    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3261e1-75b769cd58-wcvz5"]
Jul 29 15:27:17.648951 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:17.648905    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0311700d-2984-4e8c-aedf-6df1efab1457-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3261e1-75b769cd58-wcvz5\" (UID: \"0311700d-2984-4e8c-aedf-6df1efab1457\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3261e1-75b769cd58-wcvz5"
Jul 29 15:27:17.648951 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:17.648960    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhjdz\" (UniqueName: \"kubernetes.io/projected/0311700d-2984-4e8c-aedf-6df1efab1457-kube-api-access-vhjdz\") pod \"maas-api-e2e-ipp-a-3261e1-75b769cd58-wcvz5\" (UID: \"0311700d-2984-4e8c-aedf-6df1efab1457\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3261e1-75b769cd58-wcvz5"
Jul 29 15:27:17.649221 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:17.649100    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0311700d-2984-4e8c-aedf-6df1efab1457-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3261e1-75b769cd58-wcvz5\" (UID: \"0311700d-2984-4e8c-aedf-6df1efab1457\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3261e1-75b769cd58-wcvz5"
Jul 29 15:27:17.749681 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:17.749638    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0311700d-2984-4e8c-aedf-6df1efab1457-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3261e1-75b769cd58-wcvz5\" (UID: \"0311700d-2984-4e8c-aedf-6df1efab1457\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3261e1-75b769cd58-wcvz5"
Jul 29 15:27:17.749681 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:17.749681    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhjdz\" (UniqueName: \"kubernetes.io/projected/0311700d-2984-4e8c-aedf-6df1efab1457-kube-api-access-vhjdz\") pod \"maas-api-e2e-ipp-a-3261e1-75b769cd58-wcvz5\" (UID: \"0311700d-2984-4e8c-aedf-6df1efab1457\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3261e1-75b769cd58-wcvz5"
Jul 29 15:27:17.749916 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:17.749734    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0311700d-2984-4e8c-aedf-6df1efab1457-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3261e1-75b769cd58-wcvz5\" (UID: \"0311700d-2984-4e8c-aedf-6df1efab1457\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3261e1-75b769cd58-wcvz5"
Jul 29 15:27:17.750452 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:17.750425    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0311700d-2984-4e8c-aedf-6df1efab1457-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3261e1-75b769cd58-wcvz5\" (UID: \"0311700d-2984-4e8c-aedf-6df1efab1457\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3261e1-75b769cd58-wcvz5"
Jul 29 15:27:17.752200 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:17.752169    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0311700d-2984-4e8c-aedf-6df1efab1457-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3261e1-75b769cd58-wcvz5\" (UID: \"0311700d-2984-4e8c-aedf-6df1efab1457\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3261e1-75b769cd58-wcvz5"
Jul 29 15:27:17.759840 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:17.759807    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhjdz\" (UniqueName: \"kubernetes.io/projected/0311700d-2984-4e8c-aedf-6df1efab1457-kube-api-access-vhjdz\") pod \"maas-api-e2e-ipp-a-3261e1-75b769cd58-wcvz5\" (UID: \"0311700d-2984-4e8c-aedf-6df1efab1457\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3261e1-75b769cd58-wcvz5"
Jul 29 15:27:17.842371 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:17.842267    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3261e1-75b769cd58-wcvz5"
Jul 29 15:27:17.988301 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:17.988265    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3261e1-75b769cd58-wcvz5"]
Jul 29 15:27:17.991003 ip-10-0-136-196 kubenswrapper[2563]: W0729 15:27:17.990972    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0311700d_2984_4e8c_aedf_6df1efab1457.slice/crio-01ab16c25ded74f28aa2c3ea1170f29ddfbc4304f8b16ddc7dc4ff94b44dbe94 WatchSource:0}: Error finding container 01ab16c25ded74f28aa2c3ea1170f29ddfbc4304f8b16ddc7dc4ff94b44dbe94: Status 404 returned error can't find the container with id 01ab16c25ded74f28aa2c3ea1170f29ddfbc4304f8b16ddc7dc4ff94b44dbe94
Jul 29 15:27:18.465645 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:18.465599    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3261e1-75b769cd58-wcvz5" event={"ID":"0311700d-2984-4e8c-aedf-6df1efab1457","Type":"ContainerStarted","Data":"01ab16c25ded74f28aa2c3ea1170f29ddfbc4304f8b16ddc7dc4ff94b44dbe94"}
Jul 29 15:27:19.472676 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:19.472633    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3261e1-75b769cd58-wcvz5" event={"ID":"0311700d-2984-4e8c-aedf-6df1efab1457","Type":"ContainerStarted","Data":"f3436b04ac6a68e124260ed1140a734ee9e2a165b2a8c34c3b5e30c13d0dd799"}
Jul 29 15:27:19.473198 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:19.472742    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3261e1-75b769cd58-wcvz5"
Jul 29 15:27:19.501254 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:19.501190    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3261e1-75b769cd58-wcvz5" podStartSLOduration=1.818684784 podStartE2EDuration="2.501173234s" totalImagesPullingTime="682.48845ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:27:19.499383124 +0000 UTC m=+3141.547841066" watchObservedRunningTime="2026-07-29 15:27:19.501173234 +0000 UTC m=+3141.549631174"
Jul 29 15:27:25.482740 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:25.482650    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3261e1-75b769cd58-wcvz5"
Jul 29 15:27:32.496310 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:32.496269    2563 patch_prober.go:28] interesting pod/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 29 15:27:32.496785 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:32.496343    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z" podUID="6746c008-6c97-4473-909b-c5c9e4c90b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 29 15:27:33.545345 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:33.545311    2563 generic.go:356] "Generic (PLEG): container finished" podID="6746c008-6c97-4473-909b-c5c9e4c90b7a" containerID="a1ed64ca1ab6d14da167d8a6adf975af1cb3ccda33685c158d2384595e3e5a05" exitCode=0
Jul 29 15:27:33.545796 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:33.545382    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z" event={"ID":"6746c008-6c97-4473-909b-c5c9e4c90b7a","Type":"ContainerDied","Data":"a1ed64ca1ab6d14da167d8a6adf975af1cb3ccda33685c158d2384595e3e5a05"}
Jul 29 15:27:33.545856 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:33.545837    2563 scope.go:117] "RemoveContainer" containerID="a1ed64ca1ab6d14da167d8a6adf975af1cb3ccda33685c158d2384595e3e5a05"
Jul 29 15:27:33.546990 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:33.546962    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:27:33.547101 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:33.547048    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:27:33.547101 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:33.547094    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:27:34.557529 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:34.557492    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z" event={"ID":"6746c008-6c97-4473-909b-c5c9e4c90b7a","Type":"ContainerStarted","Data":"fb99103fb31b9d144a07907672372cbd5e01c0daed141584011a00301db1bd1c"}
Jul 29 15:27:35.505379 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:35.505337    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"
Jul 29 15:27:35.505635 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:35.505608    2563 patch_prober.go:28] interesting pod/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Jul 29 15:27:35.505691 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:35.505665    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z" podUID="6746c008-6c97-4473-909b-c5c9e4c90b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Jul 29 15:27:36.505842 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:36.505796    2563 patch_prober.go:28] interesting pod/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Jul 29 15:27:36.506388 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:36.505887    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z" podUID="6746c008-6c97-4473-909b-c5c9e4c90b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Jul 29 15:27:37.506393 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:37.506346    2563 patch_prober.go:28] interesting pod/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Jul 29 15:27:37.506811 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:37.506427    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z" podUID="6746c008-6c97-4473-909b-c5c9e4c90b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Jul 29 15:27:38.505541 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:38.505503    2563 patch_prober.go:28] interesting pod/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Jul 29 15:27:38.505747 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:38.505582    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z" podUID="6746c008-6c97-4473-909b-c5c9e4c90b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Jul 29 15:27:39.508848 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:39.508813    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"
Jul 29 15:27:39.509278 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:39.509085    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"
Jul 29 15:27:39.509832 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:39.509813    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"
Jul 29 15:27:46.413487 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:46.413444    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4092e-57db86cd8d-2mdml"]
Jul 29 15:27:46.416109 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:46.416089    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4092e-57db86cd8d-2mdml"
Jul 29 15:27:46.418179 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:46.418156    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-a4092e-serving-cert\""
Jul 29 15:27:46.433478 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:46.433437    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zbdz\" (UniqueName: \"kubernetes.io/projected/55505629-2930-4ca3-b68f-908a0f1f328c-kube-api-access-2zbdz\") pod \"maas-api-e2e-ipp-b-a4092e-57db86cd8d-2mdml\" (UID: \"55505629-2930-4ca3-b68f-908a0f1f328c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4092e-57db86cd8d-2mdml"
Jul 29 15:27:46.433644 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:46.433621    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55505629-2930-4ca3-b68f-908a0f1f328c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a4092e-57db86cd8d-2mdml\" (UID: \"55505629-2930-4ca3-b68f-908a0f1f328c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4092e-57db86cd8d-2mdml"
Jul 29 15:27:46.433698 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:46.433671    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55505629-2930-4ca3-b68f-908a0f1f328c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a4092e-57db86cd8d-2mdml\" (UID: \"55505629-2930-4ca3-b68f-908a0f1f328c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4092e-57db86cd8d-2mdml"
Jul 29 15:27:46.435161 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:46.435104    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4092e-57db86cd8d-2mdml"]
Jul 29 15:27:46.534490 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:46.534452    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zbdz\" (UniqueName: \"kubernetes.io/projected/55505629-2930-4ca3-b68f-908a0f1f328c-kube-api-access-2zbdz\") pod \"maas-api-e2e-ipp-b-a4092e-57db86cd8d-2mdml\" (UID: \"55505629-2930-4ca3-b68f-908a0f1f328c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4092e-57db86cd8d-2mdml"
Jul 29 15:27:46.534683 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:46.534537    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55505629-2930-4ca3-b68f-908a0f1f328c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a4092e-57db86cd8d-2mdml\" (UID: \"55505629-2930-4ca3-b68f-908a0f1f328c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4092e-57db86cd8d-2mdml"
Jul 29 15:27:46.534683 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:46.534564    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55505629-2930-4ca3-b68f-908a0f1f328c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a4092e-57db86cd8d-2mdml\" (UID: \"55505629-2930-4ca3-b68f-908a0f1f328c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4092e-57db86cd8d-2mdml"
Jul 29 15:27:46.534683 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:27:46.534674    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4092e-serving-cert: secret "maas-api-e2e-ipp-b-a4092e-serving-cert" not found
Jul 29 15:27:46.534796 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:27:46.534753    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55505629-2930-4ca3-b68f-908a0f1f328c-maas-api-tls podName:55505629-2930-4ca3-b68f-908a0f1f328c nodeName:}" failed. No retries permitted until 2026-07-29 15:27:47.034732087 +0000 UTC m=+3169.083190004 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/55505629-2930-4ca3-b68f-908a0f1f328c-maas-api-tls") pod "maas-api-e2e-ipp-b-a4092e-57db86cd8d-2mdml" (UID: "55505629-2930-4ca3-b68f-908a0f1f328c") : secret "maas-api-e2e-ipp-b-a4092e-serving-cert" not found
Jul 29 15:27:46.535323 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:46.535302    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55505629-2930-4ca3-b68f-908a0f1f328c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a4092e-57db86cd8d-2mdml\" (UID: \"55505629-2930-4ca3-b68f-908a0f1f328c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4092e-57db86cd8d-2mdml"
Jul 29 15:27:46.542844 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:46.542813    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zbdz\" (UniqueName: \"kubernetes.io/projected/55505629-2930-4ca3-b68f-908a0f1f328c-kube-api-access-2zbdz\") pod \"maas-api-e2e-ipp-b-a4092e-57db86cd8d-2mdml\" (UID: \"55505629-2930-4ca3-b68f-908a0f1f328c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4092e-57db86cd8d-2mdml"
Jul 29 15:27:47.039838 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:47.039797    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55505629-2930-4ca3-b68f-908a0f1f328c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a4092e-57db86cd8d-2mdml\" (UID: \"55505629-2930-4ca3-b68f-908a0f1f328c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4092e-57db86cd8d-2mdml"
Jul 29 15:27:47.042653 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:47.042625    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55505629-2930-4ca3-b68f-908a0f1f328c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a4092e-57db86cd8d-2mdml\" (UID: \"55505629-2930-4ca3-b68f-908a0f1f328c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4092e-57db86cd8d-2mdml"
Jul 29 15:27:47.327392 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:47.327294    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4092e-57db86cd8d-2mdml"
Jul 29 15:27:47.474940 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:47.474912    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4092e-57db86cd8d-2mdml"]
Jul 29 15:27:47.477615 ip-10-0-136-196 kubenswrapper[2563]: W0729 15:27:47.477586    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55505629_2930_4ca3_b68f_908a0f1f328c.slice/crio-107f23307eb8493ceac1addf724c3635b4f1541fae599f305748d6adf478a0d6 WatchSource:0}: Error finding container 107f23307eb8493ceac1addf724c3635b4f1541fae599f305748d6adf478a0d6: Status 404 returned error can't find the container with id 107f23307eb8493ceac1addf724c3635b4f1541fae599f305748d6adf478a0d6
Jul 29 15:27:47.637277 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:47.637186    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4092e-57db86cd8d-2mdml" event={"ID":"55505629-2930-4ca3-b68f-908a0f1f328c","Type":"ContainerStarted","Data":"107f23307eb8493ceac1addf724c3635b4f1541fae599f305748d6adf478a0d6"}
Jul 29 15:27:48.643469 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:48.643432    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4092e-57db86cd8d-2mdml" event={"ID":"55505629-2930-4ca3-b68f-908a0f1f328c","Type":"ContainerStarted","Data":"b537cbc34c0550dd584af02b29d9983522c6fce6d8570f4cad0e39fb79222109"}
Jul 29 15:27:48.643874 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:48.643558    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4092e-57db86cd8d-2mdml"
Jul 29 15:27:48.660114 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:48.660050    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4092e-57db86cd8d-2mdml" podStartSLOduration=2.312590285 podStartE2EDuration="2.660031984s" totalImagesPullingTime="347.441699ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:27:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:27:48.659053938 +0000 UTC m=+3170.707511877" watchObservedRunningTime="2026-07-29 15:27:48.660031984 +0000 UTC m=+3170.708489924"
Jul 29 15:27:54.654333 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:27:54.654305    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4092e-57db86cd8d-2mdml"
Jul 29 15:29:09.249602 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:09.249551    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"]
Jul 29 15:29:09.253784 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:09.253756    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"
Jul 29 15:29:09.255888 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:09.255865    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-04773a-openshift-default-dockercfg-s4qm4\""
Jul 29 15:29:09.264358 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:09.264044    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"]
Jul 29 15:29:09.324654 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:09.324603    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b7550d2-d2f2-434b-a671-4ffd6d32f253-workload-certs\") pod \"e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp\" (UID: \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\") " pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"
Jul 29 15:29:09.324834 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:09.324668    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b7550d2-d2f2-434b-a671-4ffd6d32f253-workload-socket\") pod \"e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp\" (UID: \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\") " pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"
Jul 29 15:29:09.324834 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:09.324721    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b7550d2-d2f2-434b-a671-4ffd6d32f253-istio-podinfo\") pod \"e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp\" (UID: \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\") " pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"
Jul 29 15:29:09.324834 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:09.324789    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b7550d2-d2f2-434b-a671-4ffd6d32f253-istiod-ca-cert\") pod \"e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp\" (UID: \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\") " pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"
Jul 29 15:29:09.324834 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:09.324828    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b7550d2-d2f2-434b-a671-4ffd6d32f253-istio-data\") pod \"e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp\" (UID: \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\") " pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"
Jul 29 15:29:09.324983 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:09.324863    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jtn9\" (UniqueName: \"kubernetes.io/projected/1b7550d2-d2f2-434b-a671-4ffd6d32f253-kube-api-access-7jtn9\") pod \"e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp\" (UID: \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\") " pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"
Jul 29 15:29:09.324983 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:09.324886    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b7550d2-d2f2-434b-a671-4ffd6d32f253-istio-token\") pod \"e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp\" (UID: \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\") " pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"
Jul 29 15:29:09.325048 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:09.325000    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b7550d2-d2f2-434b-a671-4ffd6d32f253-istio-envoy\") pod \"e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp\" (UID: \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\") " pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"
Jul 29 15:29:09.325086 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:09.325044    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b7550d2-d2f2-434b-a671-4ffd6d32f253-credential-socket\") pod \"e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp\" (UID: \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\") " pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"
Jul 29 15:29:09.426178 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:09.426115    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jtn9\" (UniqueName: \"kubernetes.io/projected/1b7550d2-d2f2-434b-a671-4ffd6d32f253-kube-api-access-7jtn9\") pod \"e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp\" (UID: \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\") " pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"
Jul 29 15:29:09.426178 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:09.426182    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b7550d2-d2f2-434b-a671-4ffd6d32f253-istio-token\") pod \"e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp\" (UID: \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\") " pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"
Jul 29 15:29:09.426461 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:09.426221    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b7550d2-d2f2-434b-a671-4ffd6d32f253-istio-envoy\") pod \"e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp\" (UID: \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\") " pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"
Jul 29 15:29:09.426461 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:09.426242    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b7550d2-d2f2-434b-a671-4ffd6d32f253-credential-socket\") pod \"e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp\" (UID: \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\") " pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"
Jul 29 15:29:09.426461 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:09.426278    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b7550d2-d2f2-434b-a671-4ffd6d32f253-workload-certs\") pod \"e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp\" (UID: \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\") " pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"
Jul 29 15:29:09.426461 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:09.426323    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b7550d2-d2f2-434b-a671-4ffd6d32f253-workload-socket\") pod \"e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp\" (UID: \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\") " pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"
Jul 29 15:29:09.426461 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:09.426344    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b7550d2-d2f2-434b-a671-4ffd6d32f253-istio-podinfo\") pod \"e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp\" (UID: \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\") " pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"
Jul 29 15:29:09.426724 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:09.426504    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b7550d2-d2f2-434b-a671-4ffd6d32f253-istiod-ca-cert\") pod \"e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp\" (UID: \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\") " pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"
Jul 29 15:29:09.426724 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:09.426570    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b7550d2-d2f2-434b-a671-4ffd6d32f253-istio-data\") pod \"e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp\" (UID: \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\") " pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"
Jul 29 15:29:09.426838 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:09.426775    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b7550d2-d2f2-434b-a671-4ffd6d32f253-credential-socket\") pod \"e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp\" (UID: \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\") " pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"
Jul 29 15:29:09.426898 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:09.426880    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b7550d2-d2f2-434b-a671-4ffd6d32f253-workload-socket\") pod \"e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp\" (UID: \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\") " pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"
Jul 29 15:29:09.427043 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:09.427017    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b7550d2-d2f2-434b-a671-4ffd6d32f253-istio-data\") pod \"e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp\" (UID: \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\") " pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"
Jul 29 15:29:09.427185 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:09.427071    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b7550d2-d2f2-434b-a671-4ffd6d32f253-workload-certs\") pod \"e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp\" (UID: \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\") " pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"
Jul 29 15:29:09.427384 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:09.427357    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b7550d2-d2f2-434b-a671-4ffd6d32f253-istiod-ca-cert\") pod \"e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp\" (UID: \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\") " pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"
Jul 29 15:29:09.428673 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:09.428648    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b7550d2-d2f2-434b-a671-4ffd6d32f253-istio-envoy\") pod \"e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp\" (UID: \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\") " pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"
Jul 29 15:29:09.429072 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:09.429055    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b7550d2-d2f2-434b-a671-4ffd6d32f253-istio-podinfo\") pod \"e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp\" (UID: \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\") " pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"
Jul 29 15:29:09.435161 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:09.435110    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b7550d2-d2f2-434b-a671-4ffd6d32f253-istio-token\") pod \"e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp\" (UID: \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\") " pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"
Jul 29 15:29:09.435332 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:09.435309    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jtn9\" (UniqueName: \"kubernetes.io/projected/1b7550d2-d2f2-434b-a671-4ffd6d32f253-kube-api-access-7jtn9\") pod \"e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp\" (UID: \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\") " pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"
Jul 29 15:29:09.568902 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:09.568803    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"
Jul 29 15:29:09.721618 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:09.721569    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"]
Jul 29 15:29:09.723965 ip-10-0-136-196 kubenswrapper[2563]: W0729 15:29:09.723941    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b7550d2_d2f2_434b_a671_4ffd6d32f253.slice/crio-7b9ebc831145b0b104907f6ff5a701c7760f938bc3565e5a711ace3f5da62531 WatchSource:0}: Error finding container 7b9ebc831145b0b104907f6ff5a701c7760f938bc3565e5a711ace3f5da62531: Status 404 returned error can't find the container with id 7b9ebc831145b0b104907f6ff5a701c7760f938bc3565e5a711ace3f5da62531
Jul 29 15:29:09.726063 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:09.726031    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:29:09.726151 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:09.726098    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:29:09.726205 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:09.726126    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:29:10.085657 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:10.085619    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp" event={"ID":"1b7550d2-d2f2-434b-a671-4ffd6d32f253","Type":"ContainerStarted","Data":"dca17910382194cbba9b339582c14dffb80e4b792819299cc377aef14bd19611"}
Jul 29 15:29:10.085871 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:10.085670    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp" event={"ID":"1b7550d2-d2f2-434b-a671-4ffd6d32f253","Type":"ContainerStarted","Data":"7b9ebc831145b0b104907f6ff5a701c7760f938bc3565e5a711ace3f5da62531"}
Jul 29 15:29:10.106153 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:10.106074    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp" podStartSLOduration=1.10605283 podStartE2EDuration="1.10605283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:29:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:29:10.103850075 +0000 UTC m=+3252.152308015" watchObservedRunningTime="2026-07-29 15:29:10.10605283 +0000 UTC m=+3252.154510792"
Jul 29 15:29:10.569558 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:10.569517    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"
Jul 29 15:29:10.574921 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:10.574892    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"
Jul 29 15:29:11.090651 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:11.090609    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"
Jul 29 15:29:11.091726 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:11.091705    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"
Jul 29 15:29:12.126885 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:12.126839    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-04773a-76ff59b595-62cm5"]
Jul 29 15:29:12.130847 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:12.130820    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-04773a-76ff59b595-62cm5"
Jul 29 15:29:12.133568 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:12.133542    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-04773a-serving-cert\""
Jul 29 15:29:12.141399 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:12.141371    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-04773a-76ff59b595-62cm5"]
Jul 29 15:29:12.155466 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:12.155426    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g9hh\" (UniqueName: \"kubernetes.io/projected/3776b8ec-0c4f-486c-b7e7-307e0c679c38-kube-api-access-7g9hh\") pod \"maas-api-e2e-ipp-cleanup-04773a-76ff59b595-62cm5\" (UID: \"3776b8ec-0c4f-486c-b7e7-307e0c679c38\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-04773a-76ff59b595-62cm5"
Jul 29 15:29:12.155744 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:12.155716    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3776b8ec-0c4f-486c-b7e7-307e0c679c38-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-04773a-76ff59b595-62cm5\" (UID: \"3776b8ec-0c4f-486c-b7e7-307e0c679c38\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-04773a-76ff59b595-62cm5"
Jul 29 15:29:12.155873 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:12.155796    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3776b8ec-0c4f-486c-b7e7-307e0c679c38-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-04773a-76ff59b595-62cm5\" (UID: \"3776b8ec-0c4f-486c-b7e7-307e0c679c38\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-04773a-76ff59b595-62cm5"
Jul 29 15:29:12.256497 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:12.256456    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g9hh\" (UniqueName: \"kubernetes.io/projected/3776b8ec-0c4f-486c-b7e7-307e0c679c38-kube-api-access-7g9hh\") pod \"maas-api-e2e-ipp-cleanup-04773a-76ff59b595-62cm5\" (UID: \"3776b8ec-0c4f-486c-b7e7-307e0c679c38\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-04773a-76ff59b595-62cm5"
Jul 29 15:29:12.256696 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:12.256518    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3776b8ec-0c4f-486c-b7e7-307e0c679c38-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-04773a-76ff59b595-62cm5\" (UID: \"3776b8ec-0c4f-486c-b7e7-307e0c679c38\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-04773a-76ff59b595-62cm5"
Jul 29 15:29:12.256696 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:12.256578    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3776b8ec-0c4f-486c-b7e7-307e0c679c38-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-04773a-76ff59b595-62cm5\" (UID: \"3776b8ec-0c4f-486c-b7e7-307e0c679c38\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-04773a-76ff59b595-62cm5"
Jul 29 15:29:12.256789 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:29:12.256694    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-04773a-serving-cert: secret "maas-api-e2e-ipp-cleanup-04773a-serving-cert" not found
Jul 29 15:29:12.256789 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:29:12.256781    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3776b8ec-0c4f-486c-b7e7-307e0c679c38-maas-api-tls podName:3776b8ec-0c4f-486c-b7e7-307e0c679c38 nodeName:}" failed. No retries permitted until 2026-07-29 15:29:12.756760671 +0000 UTC m=+3254.805218616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3776b8ec-0c4f-486c-b7e7-307e0c679c38-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-04773a-76ff59b595-62cm5" (UID: "3776b8ec-0c4f-486c-b7e7-307e0c679c38") : secret "maas-api-e2e-ipp-cleanup-04773a-serving-cert" not found
Jul 29 15:29:12.257374 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:12.257354    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3776b8ec-0c4f-486c-b7e7-307e0c679c38-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-04773a-76ff59b595-62cm5\" (UID: \"3776b8ec-0c4f-486c-b7e7-307e0c679c38\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-04773a-76ff59b595-62cm5"
Jul 29 15:29:12.264961 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:12.264924    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g9hh\" (UniqueName: \"kubernetes.io/projected/3776b8ec-0c4f-486c-b7e7-307e0c679c38-kube-api-access-7g9hh\") pod \"maas-api-e2e-ipp-cleanup-04773a-76ff59b595-62cm5\" (UID: \"3776b8ec-0c4f-486c-b7e7-307e0c679c38\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-04773a-76ff59b595-62cm5"
Jul 29 15:29:12.761048 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:12.761007    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3776b8ec-0c4f-486c-b7e7-307e0c679c38-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-04773a-76ff59b595-62cm5\" (UID: \"3776b8ec-0c4f-486c-b7e7-307e0c679c38\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-04773a-76ff59b595-62cm5"
Jul 29 15:29:12.763728 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:12.763695    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3776b8ec-0c4f-486c-b7e7-307e0c679c38-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-04773a-76ff59b595-62cm5\" (UID: \"3776b8ec-0c4f-486c-b7e7-307e0c679c38\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-04773a-76ff59b595-62cm5"
Jul 29 15:29:13.044477 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:13.044381    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-04773a-76ff59b595-62cm5"
Jul 29 15:29:13.204188 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:13.204120    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-04773a-76ff59b595-62cm5"]
Jul 29 15:29:13.207795 ip-10-0-136-196 kubenswrapper[2563]: W0729 15:29:13.207757    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3776b8ec_0c4f_486c_b7e7_307e0c679c38.slice/crio-f02691c66176c3ca396c79ba944d8c5d5496e68300979e906e4175d3b09f6155 WatchSource:0}: Error finding container f02691c66176c3ca396c79ba944d8c5d5496e68300979e906e4175d3b09f6155: Status 404 returned error can't find the container with id f02691c66176c3ca396c79ba944d8c5d5496e68300979e906e4175d3b09f6155
Jul 29 15:29:14.117575 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:14.117529    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-04773a-76ff59b595-62cm5" event={"ID":"3776b8ec-0c4f-486c-b7e7-307e0c679c38","Type":"ContainerStarted","Data":"0aa31b950f87ae2c7940a662ab8179ed7b10538f25849f873c9af141a399869d"}
Jul 29 15:29:14.117776 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:14.117628    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-04773a-76ff59b595-62cm5"
Jul 29 15:29:14.117776 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:14.117650    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-04773a-76ff59b595-62cm5" event={"ID":"3776b8ec-0c4f-486c-b7e7-307e0c679c38","Type":"ContainerStarted","Data":"f02691c66176c3ca396c79ba944d8c5d5496e68300979e906e4175d3b09f6155"}
Jul 29 15:29:20.128096 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:20.128064    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-04773a-76ff59b595-62cm5"
Jul 29 15:29:20.145831 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:20.145754    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-04773a-76ff59b595-62cm5" podStartSLOduration=7.802736201 podStartE2EDuration="8.145732193s" totalImagesPullingTime="342.995992ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:29:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:29:14.133117984 +0000 UTC m=+3256.181575926" watchObservedRunningTime="2026-07-29 15:29:20.145732193 +0000 UTC m=+3262.194190133"
Jul 29 15:29:26.092704 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:26.092667    2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Jul 29 15:29:26.093109 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:26.092726    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp" podUID="1b7550d2-d2f2-434b-a671-4ffd6d32f253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Jul 29 15:29:26.191777 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:26.191697    2563 generic.go:356] "Generic (PLEG): container finished" podID="1b7550d2-d2f2-434b-a671-4ffd6d32f253" containerID="dca17910382194cbba9b339582c14dffb80e4b792819299cc377aef14bd19611" exitCode=0
Jul 29 15:29:26.191777 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:26.191737    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp" event={"ID":"1b7550d2-d2f2-434b-a671-4ffd6d32f253","Type":"ContainerDied","Data":"dca17910382194cbba9b339582c14dffb80e4b792819299cc377aef14bd19611"}
Jul 29 15:29:26.192174 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:26.192158    2563 scope.go:117] "RemoveContainer" containerID="dca17910382194cbba9b339582c14dffb80e4b792819299cc377aef14bd19611"
Jul 29 15:29:26.193241 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:26.193213    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:29:26.193359 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:26.193280    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:29:26.193359 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:26.193309    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 15:29:27.199050 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:27.199012    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp" event={"ID":"1b7550d2-d2f2-434b-a671-4ffd6d32f253","Type":"ContainerStarted","Data":"203bd2216287908950d04ecff54bcbd994d3af7ea5b49ed245f55f25d749b43a"}
Jul 29 15:29:27.569874 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:27.569836    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"
Jul 29 15:29:27.570317 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:27.570219    2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Jul 29 15:29:27.570481 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:27.570344    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp" podUID="1b7550d2-d2f2-434b-a671-4ffd6d32f253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Jul 29 15:29:28.570224 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:28.570177    2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Jul 29 15:29:28.570668 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:28.570245    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp" podUID="1b7550d2-d2f2-434b-a671-4ffd6d32f253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Jul 29 15:29:29.569880 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:29.569823    2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Jul 29 15:29:29.570107 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:29.569899    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp" podUID="1b7550d2-d2f2-434b-a671-4ffd6d32f253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Jul 29 15:29:30.569630 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:30.569573    2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Jul 29 15:29:30.570175 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:30.569666    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp" podUID="1b7550d2-d2f2-434b-a671-4ffd6d32f253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Jul 29 15:29:31.573940 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:31.573898    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"
Jul 29 15:29:31.574356 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:31.574169    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"
Jul 29 15:29:31.574936 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:31.574919    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"
Jul 29 15:29:42.864224 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:42.864185    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-04773a-76ff59b595-62cm5"]
Jul 29 15:29:42.864773 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:42.864407    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-04773a-76ff59b595-62cm5" podUID="3776b8ec-0c4f-486c-b7e7-307e0c679c38" containerName="maas-api" containerID="cri-o://0aa31b950f87ae2c7940a662ab8179ed7b10538f25849f873c9af141a399869d" gracePeriod=30
Jul 29 15:29:43.139814 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:43.139785    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-04773a-76ff59b595-62cm5"
Jul 29 15:29:43.176612 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:43.176577    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3776b8ec-0c4f-486c-b7e7-307e0c679c38-maas-api-tls\") pod \"3776b8ec-0c4f-486c-b7e7-307e0c679c38\" (UID: \"3776b8ec-0c4f-486c-b7e7-307e0c679c38\") "
Jul 29 15:29:43.176800 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:43.176701    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3776b8ec-0c4f-486c-b7e7-307e0c679c38-openshift-service-ca\") pod \"3776b8ec-0c4f-486c-b7e7-307e0c679c38\" (UID: \"3776b8ec-0c4f-486c-b7e7-307e0c679c38\") "
Jul 29 15:29:43.176800 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:43.176762    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7g9hh\" (UniqueName: \"kubernetes.io/projected/3776b8ec-0c4f-486c-b7e7-307e0c679c38-kube-api-access-7g9hh\") pod \"3776b8ec-0c4f-486c-b7e7-307e0c679c38\" (UID: \"3776b8ec-0c4f-486c-b7e7-307e0c679c38\") "
Jul 29 15:29:43.177128 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:43.177099    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3776b8ec-0c4f-486c-b7e7-307e0c679c38-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3776b8ec-0c4f-486c-b7e7-307e0c679c38" (UID: "3776b8ec-0c4f-486c-b7e7-307e0c679c38"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 15:29:43.178815 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:43.178789    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3776b8ec-0c4f-486c-b7e7-307e0c679c38-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3776b8ec-0c4f-486c-b7e7-307e0c679c38" (UID: "3776b8ec-0c4f-486c-b7e7-307e0c679c38"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 15:29:43.178903 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:43.178819    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3776b8ec-0c4f-486c-b7e7-307e0c679c38-kube-api-access-7g9hh" (OuterVolumeSpecName: "kube-api-access-7g9hh") pod "3776b8ec-0c4f-486c-b7e7-307e0c679c38" (UID: "3776b8ec-0c4f-486c-b7e7-307e0c679c38"). InnerVolumeSpecName "kube-api-access-7g9hh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 15:29:43.277912 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:43.277874    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3776b8ec-0c4f-486c-b7e7-307e0c679c38-openshift-service-ca\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:29:43.277912 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:43.277907    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g9hh\" (UniqueName: \"kubernetes.io/projected/3776b8ec-0c4f-486c-b7e7-307e0c679c38-kube-api-access-7g9hh\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:29:43.278124 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:43.277933    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3776b8ec-0c4f-486c-b7e7-307e0c679c38-maas-api-tls\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:29:43.285274 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:43.285240    2563 generic.go:356] "Generic (PLEG): container finished" podID="3776b8ec-0c4f-486c-b7e7-307e0c679c38" containerID="0aa31b950f87ae2c7940a662ab8179ed7b10538f25849f873c9af141a399869d" exitCode=0
Jul 29 15:29:43.285415 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:43.285291    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-04773a-76ff59b595-62cm5"
Jul 29 15:29:43.285415 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:43.285327    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-04773a-76ff59b595-62cm5" event={"ID":"3776b8ec-0c4f-486c-b7e7-307e0c679c38","Type":"ContainerDied","Data":"0aa31b950f87ae2c7940a662ab8179ed7b10538f25849f873c9af141a399869d"}
Jul 29 15:29:43.285415 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:43.285382    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-04773a-76ff59b595-62cm5" event={"ID":"3776b8ec-0c4f-486c-b7e7-307e0c679c38","Type":"ContainerDied","Data":"f02691c66176c3ca396c79ba944d8c5d5496e68300979e906e4175d3b09f6155"}
Jul 29 15:29:43.285415 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:43.285404    2563 scope.go:117] "RemoveContainer" containerID="0aa31b950f87ae2c7940a662ab8179ed7b10538f25849f873c9af141a399869d"
Jul 29 15:29:43.295467 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:43.295443    2563 scope.go:117] "RemoveContainer" containerID="0aa31b950f87ae2c7940a662ab8179ed7b10538f25849f873c9af141a399869d"
Jul 29 15:29:43.295727 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:29:43.295707    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aa31b950f87ae2c7940a662ab8179ed7b10538f25849f873c9af141a399869d\": container with ID starting with 0aa31b950f87ae2c7940a662ab8179ed7b10538f25849f873c9af141a399869d not found: ID does not exist" containerID="0aa31b950f87ae2c7940a662ab8179ed7b10538f25849f873c9af141a399869d"
Jul 29 15:29:43.295778 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:43.295737    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aa31b950f87ae2c7940a662ab8179ed7b10538f25849f873c9af141a399869d"} err="failed to get container status \"0aa31b950f87ae2c7940a662ab8179ed7b10538f25849f873c9af141a399869d\": rpc error: code = NotFound desc = could not find container \"0aa31b950f87ae2c7940a662ab8179ed7b10538f25849f873c9af141a399869d\": container with ID starting with 0aa31b950f87ae2c7940a662ab8179ed7b10538f25849f873c9af141a399869d not found: ID does not exist"
Jul 29 15:29:43.309226 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:43.309193    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-04773a-76ff59b595-62cm5"]
Jul 29 15:29:43.313429 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:43.313399    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-04773a-76ff59b595-62cm5"]
Jul 29 15:29:44.432887 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:44.432844    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3776b8ec-0c4f-486c-b7e7-307e0c679c38" path="/var/lib/kubelet/pods/3776b8ec-0c4f-486c-b7e7-307e0c679c38/volumes"
Jul 29 15:29:49.628479 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:49.628443    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"]
Jul 29 15:29:49.629012 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:49.628650    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp" podUID="1b7550d2-d2f2-434b-a671-4ffd6d32f253" containerName="istio-proxy" containerID="cri-o://203bd2216287908950d04ecff54bcbd994d3af7ea5b49ed245f55f25d749b43a" gracePeriod=30
Jul 29 15:29:54.895376 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:54.895307    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"
Jul 29 15:29:54.996692 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:54.996654    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b7550d2-d2f2-434b-a671-4ffd6d32f253-istio-podinfo\") pod \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\" (UID: \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\") "
Jul 29 15:29:54.996873 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:54.996703    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b7550d2-d2f2-434b-a671-4ffd6d32f253-istiod-ca-cert\") pod \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\" (UID: \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\") "
Jul 29 15:29:54.996873 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:54.996744    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b7550d2-d2f2-434b-a671-4ffd6d32f253-istio-envoy\") pod \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\" (UID: \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\") "
Jul 29 15:29:54.996873 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:54.996793    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b7550d2-d2f2-434b-a671-4ffd6d32f253-workload-socket\") pod \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\" (UID: \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\") "
Jul 29 15:29:54.996873 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:54.996819    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b7550d2-d2f2-434b-a671-4ffd6d32f253-credential-socket\") pod \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\" (UID: \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\") "
Jul 29 15:29:54.996873 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:54.996858    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b7550d2-d2f2-434b-a671-4ffd6d32f253-istio-token\") pod \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\" (UID: \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\") "
Jul 29 15:29:54.997346 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:54.996912    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b7550d2-d2f2-434b-a671-4ffd6d32f253-workload-certs\") pod \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\" (UID: \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\") "
Jul 29 15:29:54.997346 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:54.996946    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jtn9\" (UniqueName: \"kubernetes.io/projected/1b7550d2-d2f2-434b-a671-4ffd6d32f253-kube-api-access-7jtn9\") pod \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\" (UID: \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\") "
Jul 29 15:29:54.997346 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:54.996989    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b7550d2-d2f2-434b-a671-4ffd6d32f253-istio-data\") pod \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\" (UID: \"1b7550d2-d2f2-434b-a671-4ffd6d32f253\") "
Jul 29 15:29:54.997346 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:54.997203    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b7550d2-d2f2-434b-a671-4ffd6d32f253-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1b7550d2-d2f2-434b-a671-4ffd6d32f253" (UID: "1b7550d2-d2f2-434b-a671-4ffd6d32f253"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 15:29:54.997346 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:54.997199    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7550d2-d2f2-434b-a671-4ffd6d32f253-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1b7550d2-d2f2-434b-a671-4ffd6d32f253" (UID: "1b7550d2-d2f2-434b-a671-4ffd6d32f253"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:29:54.997346 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:54.997210    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7550d2-d2f2-434b-a671-4ffd6d32f253-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1b7550d2-d2f2-434b-a671-4ffd6d32f253" (UID: "1b7550d2-d2f2-434b-a671-4ffd6d32f253"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:29:54.997346 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:54.997301    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7550d2-d2f2-434b-a671-4ffd6d32f253-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1b7550d2-d2f2-434b-a671-4ffd6d32f253" (UID: "1b7550d2-d2f2-434b-a671-4ffd6d32f253"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:29:54.997682 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:54.997420    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b7550d2-d2f2-434b-a671-4ffd6d32f253-istiod-ca-cert\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:29:54.997682 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:54.997438    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b7550d2-d2f2-434b-a671-4ffd6d32f253-workload-socket\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:29:54.997682 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:54.997451    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b7550d2-d2f2-434b-a671-4ffd6d32f253-credential-socket\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:29:54.997682 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:54.997467    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b7550d2-d2f2-434b-a671-4ffd6d32f253-workload-certs\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:29:54.998151 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:54.998106    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7550d2-d2f2-434b-a671-4ffd6d32f253-istio-data" (OuterVolumeSpecName: "istio-data") pod "1b7550d2-d2f2-434b-a671-4ffd6d32f253" (UID: "1b7550d2-d2f2-434b-a671-4ffd6d32f253"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:29:54.999227 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:54.999198    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b7550d2-d2f2-434b-a671-4ffd6d32f253-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1b7550d2-d2f2-434b-a671-4ffd6d32f253" (UID: "1b7550d2-d2f2-434b-a671-4ffd6d32f253"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 15:29:54.999343 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:54.999257    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7550d2-d2f2-434b-a671-4ffd6d32f253-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1b7550d2-d2f2-434b-a671-4ffd6d32f253" (UID: "1b7550d2-d2f2-434b-a671-4ffd6d32f253"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:29:54.999343 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:54.999266    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b7550d2-d2f2-434b-a671-4ffd6d32f253-istio-token" (OuterVolumeSpecName: "istio-token") pod "1b7550d2-d2f2-434b-a671-4ffd6d32f253" (UID: "1b7550d2-d2f2-434b-a671-4ffd6d32f253"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 15:29:54.999540 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:54.999510    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b7550d2-d2f2-434b-a671-4ffd6d32f253-kube-api-access-7jtn9" (OuterVolumeSpecName: "kube-api-access-7jtn9") pod "1b7550d2-d2f2-434b-a671-4ffd6d32f253" (UID: "1b7550d2-d2f2-434b-a671-4ffd6d32f253"). InnerVolumeSpecName "kube-api-access-7jtn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 15:29:55.098413 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:55.098369    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b7550d2-d2f2-434b-a671-4ffd6d32f253-istio-token\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:29:55.098413 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:55.098406    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jtn9\" (UniqueName: \"kubernetes.io/projected/1b7550d2-d2f2-434b-a671-4ffd6d32f253-kube-api-access-7jtn9\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:29:55.098413 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:55.098416    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b7550d2-d2f2-434b-a671-4ffd6d32f253-istio-data\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:29:55.098660 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:55.098427    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b7550d2-d2f2-434b-a671-4ffd6d32f253-istio-podinfo\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:29:55.098660 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:55.098437    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b7550d2-d2f2-434b-a671-4ffd6d32f253-istio-envoy\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:29:55.351042 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:55.351008    2563 generic.go:356] "Generic (PLEG): container finished" podID="1b7550d2-d2f2-434b-a671-4ffd6d32f253" containerID="203bd2216287908950d04ecff54bcbd994d3af7ea5b49ed245f55f25d749b43a" exitCode=0
Jul 29 15:29:55.351240 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:55.351059    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"
Jul 29 15:29:55.351240 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:55.351090    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp" event={"ID":"1b7550d2-d2f2-434b-a671-4ffd6d32f253","Type":"ContainerDied","Data":"203bd2216287908950d04ecff54bcbd994d3af7ea5b49ed245f55f25d749b43a"}
Jul 29 15:29:55.351240 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:55.351172    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp" event={"ID":"1b7550d2-d2f2-434b-a671-4ffd6d32f253","Type":"ContainerDied","Data":"7b9ebc831145b0b104907f6ff5a701c7760f938bc3565e5a711ace3f5da62531"}
Jul 29 15:29:55.351240 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:55.351205    2563 scope.go:117] "RemoveContainer" containerID="203bd2216287908950d04ecff54bcbd994d3af7ea5b49ed245f55f25d749b43a"
Jul 29 15:29:55.361097 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:55.361071    2563 scope.go:117] "RemoveContainer" containerID="dca17910382194cbba9b339582c14dffb80e4b792819299cc377aef14bd19611"
Jul 29 15:29:55.370493 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:55.370463    2563 scope.go:117] "RemoveContainer" containerID="203bd2216287908950d04ecff54bcbd994d3af7ea5b49ed245f55f25d749b43a"
Jul 29 15:29:55.370933 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:29:55.370909    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"203bd2216287908950d04ecff54bcbd994d3af7ea5b49ed245f55f25d749b43a\": container with ID starting with 203bd2216287908950d04ecff54bcbd994d3af7ea5b49ed245f55f25d749b43a not found: ID does not exist" containerID="203bd2216287908950d04ecff54bcbd994d3af7ea5b49ed245f55f25d749b43a"
Jul 29 15:29:55.371027 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:55.370942    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"203bd2216287908950d04ecff54bcbd994d3af7ea5b49ed245f55f25d749b43a"} err="failed to get container status \"203bd2216287908950d04ecff54bcbd994d3af7ea5b49ed245f55f25d749b43a\": rpc error: code = NotFound desc = could not find container \"203bd2216287908950d04ecff54bcbd994d3af7ea5b49ed245f55f25d749b43a\": container with ID starting with 203bd2216287908950d04ecff54bcbd994d3af7ea5b49ed245f55f25d749b43a not found: ID does not exist"
Jul 29 15:29:55.371027 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:55.370962    2563 scope.go:117] "RemoveContainer" containerID="dca17910382194cbba9b339582c14dffb80e4b792819299cc377aef14bd19611"
Jul 29 15:29:55.371253 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:29:55.371230    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dca17910382194cbba9b339582c14dffb80e4b792819299cc377aef14bd19611\": container with ID starting with dca17910382194cbba9b339582c14dffb80e4b792819299cc377aef14bd19611 not found: ID does not exist" containerID="dca17910382194cbba9b339582c14dffb80e4b792819299cc377aef14bd19611"
Jul 29 15:29:55.371301 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:55.371261    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dca17910382194cbba9b339582c14dffb80e4b792819299cc377aef14bd19611"} err="failed to get container status \"dca17910382194cbba9b339582c14dffb80e4b792819299cc377aef14bd19611\": rpc error: code = NotFound desc = could not find container \"dca17910382194cbba9b339582c14dffb80e4b792819299cc377aef14bd19611\": container with ID starting with dca17910382194cbba9b339582c14dffb80e4b792819299cc377aef14bd19611 not found: ID does not exist"
Jul 29 15:29:55.372837 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:55.372813    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"]
Jul 29 15:29:55.376753 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:55.376727    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-04773a-openshift-default-59b9f995b8-2m9xp"]
Jul 29 15:29:56.433467 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:56.433428    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b7550d2-d2f2-434b-a671-4ffd6d32f253" path="/var/lib/kubelet/pods/1b7550d2-d2f2-434b-a671-4ffd6d32f253/volumes"
Jul 29 15:29:58.933975 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:58.933945    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xnm95_23d79588-937d-4c6d-8760-a081024f608e/console-operator/1.log"
Jul 29 15:29:58.936726 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:29:58.936700    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xnm95_23d79588-937d-4c6d-8760-a081024f608e/console-operator/1.log"
Jul 29 15:30:08.757348 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:30:08.757305    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3261e1-75b769cd58-wcvz5"]
Jul 29 15:30:08.757722 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:30:08.757587    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3261e1-75b769cd58-wcvz5" podUID="0311700d-2984-4e8c-aedf-6df1efab1457" containerName="maas-api" containerID="cri-o://f3436b04ac6a68e124260ed1140a734ee9e2a165b2a8c34c3b5e30c13d0dd799" gracePeriod=30
Jul 29 15:30:09.023707 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:30:09.023675    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3261e1-75b769cd58-wcvz5"
Jul 29 15:30:09.130946 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:30:09.130897    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0311700d-2984-4e8c-aedf-6df1efab1457-openshift-service-ca\") pod \"0311700d-2984-4e8c-aedf-6df1efab1457\" (UID: \"0311700d-2984-4e8c-aedf-6df1efab1457\") "
Jul 29 15:30:09.131195 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:30:09.131002    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhjdz\" (UniqueName: \"kubernetes.io/projected/0311700d-2984-4e8c-aedf-6df1efab1457-kube-api-access-vhjdz\") pod \"0311700d-2984-4e8c-aedf-6df1efab1457\" (UID: \"0311700d-2984-4e8c-aedf-6df1efab1457\") "
Jul 29 15:30:09.131195 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:30:09.131082    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0311700d-2984-4e8c-aedf-6df1efab1457-maas-api-tls\") pod \"0311700d-2984-4e8c-aedf-6df1efab1457\" (UID: \"0311700d-2984-4e8c-aedf-6df1efab1457\") "
Jul 29 15:30:09.131401 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:30:09.131359    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0311700d-2984-4e8c-aedf-6df1efab1457-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0311700d-2984-4e8c-aedf-6df1efab1457" (UID: "0311700d-2984-4e8c-aedf-6df1efab1457"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 15:30:09.133245 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:30:09.133216    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0311700d-2984-4e8c-aedf-6df1efab1457-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0311700d-2984-4e8c-aedf-6df1efab1457" (UID: "0311700d-2984-4e8c-aedf-6df1efab1457"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 15:30:09.133332 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:30:09.133223    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0311700d-2984-4e8c-aedf-6df1efab1457-kube-api-access-vhjdz" (OuterVolumeSpecName: "kube-api-access-vhjdz") pod "0311700d-2984-4e8c-aedf-6df1efab1457" (UID: "0311700d-2984-4e8c-aedf-6df1efab1457"). InnerVolumeSpecName "kube-api-access-vhjdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 15:30:09.232640 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:30:09.232596    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhjdz\" (UniqueName: \"kubernetes.io/projected/0311700d-2984-4e8c-aedf-6df1efab1457-kube-api-access-vhjdz\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:30:09.232640 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:30:09.232635    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0311700d-2984-4e8c-aedf-6df1efab1457-maas-api-tls\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:30:09.232640 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:30:09.232645    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0311700d-2984-4e8c-aedf-6df1efab1457-openshift-service-ca\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:30:09.428011 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:30:09.427906    2563 generic.go:356] "Generic (PLEG): container finished" podID="0311700d-2984-4e8c-aedf-6df1efab1457" containerID="f3436b04ac6a68e124260ed1140a734ee9e2a165b2a8c34c3b5e30c13d0dd799" exitCode=0
Jul 29 15:30:09.428011 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:30:09.427952    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3261e1-75b769cd58-wcvz5" event={"ID":"0311700d-2984-4e8c-aedf-6df1efab1457","Type":"ContainerDied","Data":"f3436b04ac6a68e124260ed1140a734ee9e2a165b2a8c34c3b5e30c13d0dd799"}
Jul 29 15:30:09.428011 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:30:09.427955    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3261e1-75b769cd58-wcvz5"
Jul 29 15:30:09.428011 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:30:09.427995    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3261e1-75b769cd58-wcvz5" event={"ID":"0311700d-2984-4e8c-aedf-6df1efab1457","Type":"ContainerDied","Data":"01ab16c25ded74f28aa2c3ea1170f29ddfbc4304f8b16ddc7dc4ff94b44dbe94"}
Jul 29 15:30:09.428011 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:30:09.428017    2563 scope.go:117] "RemoveContainer" containerID="f3436b04ac6a68e124260ed1140a734ee9e2a165b2a8c34c3b5e30c13d0dd799"
Jul 29 15:30:09.438921 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:30:09.438899    2563 scope.go:117] "RemoveContainer" containerID="f3436b04ac6a68e124260ed1140a734ee9e2a165b2a8c34c3b5e30c13d0dd799"
Jul 29 15:30:09.439274 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:30:09.439248    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3436b04ac6a68e124260ed1140a734ee9e2a165b2a8c34c3b5e30c13d0dd799\": container with ID starting with f3436b04ac6a68e124260ed1140a734ee9e2a165b2a8c34c3b5e30c13d0dd799 not found: ID does not exist" containerID="f3436b04ac6a68e124260ed1140a734ee9e2a165b2a8c34c3b5e30c13d0dd799"
Jul 29 15:30:09.439382 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:30:09.439281    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3436b04ac6a68e124260ed1140a734ee9e2a165b2a8c34c3b5e30c13d0dd799"} err="failed to get container status \"f3436b04ac6a68e124260ed1140a734ee9e2a165b2a8c34c3b5e30c13d0dd799\": rpc error: code = NotFound desc = could not find container \"f3436b04ac6a68e124260ed1140a734ee9e2a165b2a8c34c3b5e30c13d0dd799\": container with ID starting with f3436b04ac6a68e124260ed1140a734ee9e2a165b2a8c34c3b5e30c13d0dd799 not found: ID does not exist"
Jul 29 15:30:09.452222 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:30:09.452186    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3261e1-75b769cd58-wcvz5"]
Jul 29 15:30:09.455715 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:30:09.455681    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3261e1-75b769cd58-wcvz5"]
Jul 29 15:30:10.432699 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:30:10.432657    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0311700d-2984-4e8c-aedf-6df1efab1457" path="/var/lib/kubelet/pods/0311700d-2984-4e8c-aedf-6df1efab1457/volumes"
Jul 29 15:30:59.465779 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:30:59.465739    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"]
Jul 29 15:30:59.466335 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:30:59.466023    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z" podUID="6746c008-6c97-4473-909b-c5c9e4c90b7a" containerName="istio-proxy" containerID="cri-o://fb99103fb31b9d144a07907672372cbd5e01c0daed141584011a00301db1bd1c" gracePeriod=30
Jul 29 15:31:04.708667 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:04.708633    2563 generic.go:356] "Generic (PLEG): container finished" podID="6746c008-6c97-4473-909b-c5c9e4c90b7a" containerID="fb99103fb31b9d144a07907672372cbd5e01c0daed141584011a00301db1bd1c" exitCode=0
Jul 29 15:31:04.709087 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:04.708713    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z" event={"ID":"6746c008-6c97-4473-909b-c5c9e4c90b7a","Type":"ContainerDied","Data":"fb99103fb31b9d144a07907672372cbd5e01c0daed141584011a00301db1bd1c"}
Jul 29 15:31:04.709087 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:04.708790    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z" event={"ID":"6746c008-6c97-4473-909b-c5c9e4c90b7a","Type":"ContainerDied","Data":"e1c0440a008670acf59de2f7125e5c845a937ac106c3c5f98eb55c0255082ff3"}
Jul 29 15:31:04.709087 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:04.708815    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1c0440a008670acf59de2f7125e5c845a937ac106c3c5f98eb55c0255082ff3"
Jul 29 15:31:04.709087 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:04.708835    2563 scope.go:117] "RemoveContainer" containerID="a1ed64ca1ab6d14da167d8a6adf975af1cb3ccda33685c158d2384595e3e5a05"
Jul 29 15:31:04.720572 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:31:04.720547    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1ed64ca1ab6d14da167d8a6adf975af1cb3ccda33685c158d2384595e3e5a05\": container with ID starting with a1ed64ca1ab6d14da167d8a6adf975af1cb3ccda33685c158d2384595e3e5a05 not found: ID does not exist" containerID="a1ed64ca1ab6d14da167d8a6adf975af1cb3ccda33685c158d2384595e3e5a05"
Jul 29 15:31:04.724164 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:04.724122    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"
Jul 29 15:31:04.778840 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:04.778733    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6746c008-6c97-4473-909b-c5c9e4c90b7a-credential-socket\") pod \"6746c008-6c97-4473-909b-c5c9e4c90b7a\" (UID: \"6746c008-6c97-4473-909b-c5c9e4c90b7a\") "
Jul 29 15:31:04.778840 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:04.778800    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6746c008-6c97-4473-909b-c5c9e4c90b7a-workload-certs\") pod \"6746c008-6c97-4473-909b-c5c9e4c90b7a\" (UID: \"6746c008-6c97-4473-909b-c5c9e4c90b7a\") "
Jul 29 15:31:04.778840 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:04.778828    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6746c008-6c97-4473-909b-c5c9e4c90b7a-istio-envoy\") pod \"6746c008-6c97-4473-909b-c5c9e4c90b7a\" (UID: \"6746c008-6c97-4473-909b-c5c9e4c90b7a\") "
Jul 29 15:31:04.779187 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:04.778914    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6746c008-6c97-4473-909b-c5c9e4c90b7a-istio-podinfo\") pod \"6746c008-6c97-4473-909b-c5c9e4c90b7a\" (UID: \"6746c008-6c97-4473-909b-c5c9e4c90b7a\") "
Jul 29 15:31:04.779187 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:04.778937    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6746c008-6c97-4473-909b-c5c9e4c90b7a-istio-token\") pod \"6746c008-6c97-4473-909b-c5c9e4c90b7a\" (UID: \"6746c008-6c97-4473-909b-c5c9e4c90b7a\") "
Jul 29 15:31:04.779187 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:04.778960    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6746c008-6c97-4473-909b-c5c9e4c90b7a-istio-data\") pod \"6746c008-6c97-4473-909b-c5c9e4c90b7a\" (UID: \"6746c008-6c97-4473-909b-c5c9e4c90b7a\") "
Jul 29 15:31:04.779187 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:04.778999    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzdj9\" (UniqueName: \"kubernetes.io/projected/6746c008-6c97-4473-909b-c5c9e4c90b7a-kube-api-access-tzdj9\") pod \"6746c008-6c97-4473-909b-c5c9e4c90b7a\" (UID: \"6746c008-6c97-4473-909b-c5c9e4c90b7a\") "
Jul 29 15:31:04.779187 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:04.779093    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6746c008-6c97-4473-909b-c5c9e4c90b7a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6746c008-6c97-4473-909b-c5c9e4c90b7a" (UID: "6746c008-6c97-4473-909b-c5c9e4c90b7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:31:04.779187 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:04.779110    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6746c008-6c97-4473-909b-c5c9e4c90b7a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6746c008-6c97-4473-909b-c5c9e4c90b7a" (UID: "6746c008-6c97-4473-909b-c5c9e4c90b7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:31:04.779187 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:04.779182    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6746c008-6c97-4473-909b-c5c9e4c90b7a-istiod-ca-cert\") pod \"6746c008-6c97-4473-909b-c5c9e4c90b7a\" (UID: \"6746c008-6c97-4473-909b-c5c9e4c90b7a\") "
Jul 29 15:31:04.779604 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:04.779242    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6746c008-6c97-4473-909b-c5c9e4c90b7a-workload-socket\") pod \"6746c008-6c97-4473-909b-c5c9e4c90b7a\" (UID: \"6746c008-6c97-4473-909b-c5c9e4c90b7a\") "
Jul 29 15:31:04.779604 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:04.779576    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6746c008-6c97-4473-909b-c5c9e4c90b7a-credential-socket\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:31:04.779604 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:04.779598    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6746c008-6c97-4473-909b-c5c9e4c90b7a-workload-certs\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:31:04.779778 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:04.779667    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6746c008-6c97-4473-909b-c5c9e4c90b7a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6746c008-6c97-4473-909b-c5c9e4c90b7a" (UID: "6746c008-6c97-4473-909b-c5c9e4c90b7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:31:04.779972 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:04.779951    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6746c008-6c97-4473-909b-c5c9e4c90b7a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6746c008-6c97-4473-909b-c5c9e4c90b7a" (UID: "6746c008-6c97-4473-909b-c5c9e4c90b7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 15:31:04.780050 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:04.779991    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6746c008-6c97-4473-909b-c5c9e4c90b7a-istio-data" (OuterVolumeSpecName: "istio-data") pod "6746c008-6c97-4473-909b-c5c9e4c90b7a" (UID: "6746c008-6c97-4473-909b-c5c9e4c90b7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:31:04.781335 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:04.781311    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6746c008-6c97-4473-909b-c5c9e4c90b7a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6746c008-6c97-4473-909b-c5c9e4c90b7a" (UID: "6746c008-6c97-4473-909b-c5c9e4c90b7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 15:31:04.781452 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:04.781341    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6746c008-6c97-4473-909b-c5c9e4c90b7a-kube-api-access-tzdj9" (OuterVolumeSpecName: "kube-api-access-tzdj9") pod "6746c008-6c97-4473-909b-c5c9e4c90b7a" (UID: "6746c008-6c97-4473-909b-c5c9e4c90b7a"). InnerVolumeSpecName "kube-api-access-tzdj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 15:31:04.781513 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:04.781485    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6746c008-6c97-4473-909b-c5c9e4c90b7a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6746c008-6c97-4473-909b-c5c9e4c90b7a" (UID: "6746c008-6c97-4473-909b-c5c9e4c90b7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:31:04.781680 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:04.781659    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6746c008-6c97-4473-909b-c5c9e4c90b7a-istio-token" (OuterVolumeSpecName: "istio-token") pod "6746c008-6c97-4473-909b-c5c9e4c90b7a" (UID: "6746c008-6c97-4473-909b-c5c9e4c90b7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 15:31:04.880060 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:04.880019    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzdj9\" (UniqueName: \"kubernetes.io/projected/6746c008-6c97-4473-909b-c5c9e4c90b7a-kube-api-access-tzdj9\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:31:04.880060 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:04.880052    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6746c008-6c97-4473-909b-c5c9e4c90b7a-istiod-ca-cert\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:31:04.880060 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:04.880063    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6746c008-6c97-4473-909b-c5c9e4c90b7a-workload-socket\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:31:04.880060 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:04.880072    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6746c008-6c97-4473-909b-c5c9e4c90b7a-istio-envoy\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:31:04.880377 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:04.880082    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6746c008-6c97-4473-909b-c5c9e4c90b7a-istio-podinfo\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:31:04.880377 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:04.880090    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6746c008-6c97-4473-909b-c5c9e4c90b7a-istio-token\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:31:04.880377 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:04.880097    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6746c008-6c97-4473-909b-c5c9e4c90b7a-istio-data\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:31:05.715460 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:05.715431    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"
Jul 29 15:31:05.740615 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:05.740577    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"]
Jul 29 15:31:05.750056 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:05.750022    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3261e1-openshift-default-99b6c9fc4-9hq7z"]
Jul 29 15:31:06.432996 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:06.432950    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6746c008-6c97-4473-909b-c5c9e4c90b7a" path="/var/lib/kubelet/pods/6746c008-6c97-4473-909b-c5c9e4c90b7a/volumes"
Jul 29 15:31:10.448165 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:10.448109    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4092e-57db86cd8d-2mdml"]
Jul 29 15:31:10.448636 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:10.448367    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4092e-57db86cd8d-2mdml" podUID="55505629-2930-4ca3-b68f-908a0f1f328c" containerName="maas-api" containerID="cri-o://b537cbc34c0550dd584af02b29d9983522c6fce6d8570f4cad0e39fb79222109" gracePeriod=30
Jul 29 15:31:10.715638 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:10.715602    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4092e-57db86cd8d-2mdml"
Jul 29 15:31:10.742597 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:10.742562    2563 generic.go:356] "Generic (PLEG): container finished" podID="55505629-2930-4ca3-b68f-908a0f1f328c" containerID="b537cbc34c0550dd584af02b29d9983522c6fce6d8570f4cad0e39fb79222109" exitCode=0
Jul 29 15:31:10.742776 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:10.742607    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4092e-57db86cd8d-2mdml" event={"ID":"55505629-2930-4ca3-b68f-908a0f1f328c","Type":"ContainerDied","Data":"b537cbc34c0550dd584af02b29d9983522c6fce6d8570f4cad0e39fb79222109"}
Jul 29 15:31:10.742776 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:10.742651    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4092e-57db86cd8d-2mdml" event={"ID":"55505629-2930-4ca3-b68f-908a0f1f328c","Type":"ContainerDied","Data":"107f23307eb8493ceac1addf724c3635b4f1541fae599f305748d6adf478a0d6"}
Jul 29 15:31:10.742776 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:10.742674    2563 scope.go:117] "RemoveContainer" containerID="b537cbc34c0550dd584af02b29d9983522c6fce6d8570f4cad0e39fb79222109"
Jul 29 15:31:10.742776 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:10.742613    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4092e-57db86cd8d-2mdml"
Jul 29 15:31:10.751515 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:10.751488    2563 scope.go:117] "RemoveContainer" containerID="b537cbc34c0550dd584af02b29d9983522c6fce6d8570f4cad0e39fb79222109"
Jul 29 15:31:10.751846 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:31:10.751825    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b537cbc34c0550dd584af02b29d9983522c6fce6d8570f4cad0e39fb79222109\": container with ID starting with b537cbc34c0550dd584af02b29d9983522c6fce6d8570f4cad0e39fb79222109 not found: ID does not exist" containerID="b537cbc34c0550dd584af02b29d9983522c6fce6d8570f4cad0e39fb79222109"
Jul 29 15:31:10.751918 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:10.751863    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b537cbc34c0550dd584af02b29d9983522c6fce6d8570f4cad0e39fb79222109"} err="failed to get container status \"b537cbc34c0550dd584af02b29d9983522c6fce6d8570f4cad0e39fb79222109\": rpc error: code = NotFound desc = could not find container \"b537cbc34c0550dd584af02b29d9983522c6fce6d8570f4cad0e39fb79222109\": container with ID starting with b537cbc34c0550dd584af02b29d9983522c6fce6d8570f4cad0e39fb79222109 not found: ID does not exist"
Jul 29 15:31:10.835013 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:10.834975    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zbdz\" (UniqueName: \"kubernetes.io/projected/55505629-2930-4ca3-b68f-908a0f1f328c-kube-api-access-2zbdz\") pod \"55505629-2930-4ca3-b68f-908a0f1f328c\" (UID: \"55505629-2930-4ca3-b68f-908a0f1f328c\") "
Jul 29 15:31:10.835251 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:10.835047    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55505629-2930-4ca3-b68f-908a0f1f328c-openshift-service-ca\") pod \"55505629-2930-4ca3-b68f-908a0f1f328c\" (UID: \"55505629-2930-4ca3-b68f-908a0f1f328c\") "
Jul 29 15:31:10.835251 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:10.835080    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55505629-2930-4ca3-b68f-908a0f1f328c-maas-api-tls\") pod \"55505629-2930-4ca3-b68f-908a0f1f328c\" (UID: \"55505629-2930-4ca3-b68f-908a0f1f328c\") "
Jul 29 15:31:10.835485 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:10.835462    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55505629-2930-4ca3-b68f-908a0f1f328c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "55505629-2930-4ca3-b68f-908a0f1f328c" (UID: "55505629-2930-4ca3-b68f-908a0f1f328c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 15:31:10.837358 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:10.837329    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55505629-2930-4ca3-b68f-908a0f1f328c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "55505629-2930-4ca3-b68f-908a0f1f328c" (UID: "55505629-2930-4ca3-b68f-908a0f1f328c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 15:31:10.837474 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:10.837334    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55505629-2930-4ca3-b68f-908a0f1f328c-kube-api-access-2zbdz" (OuterVolumeSpecName: "kube-api-access-2zbdz") pod "55505629-2930-4ca3-b68f-908a0f1f328c" (UID: "55505629-2930-4ca3-b68f-908a0f1f328c"). InnerVolumeSpecName "kube-api-access-2zbdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 15:31:10.936606 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:10.936545    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55505629-2930-4ca3-b68f-908a0f1f328c-openshift-service-ca\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:31:10.936606 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:10.936599    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55505629-2930-4ca3-b68f-908a0f1f328c-maas-api-tls\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:31:10.936606 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:10.936609    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zbdz\" (UniqueName: \"kubernetes.io/projected/55505629-2930-4ca3-b68f-908a0f1f328c-kube-api-access-2zbdz\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:31:11.069537 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:11.069496    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4092e-57db86cd8d-2mdml"]
Jul 29 15:31:11.073489 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:11.073457    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4092e-57db86cd8d-2mdml"]
Jul 29 15:31:12.432721 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:12.432688    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55505629-2930-4ca3-b68f-908a0f1f328c" path="/var/lib/kubelet/pods/55505629-2930-4ca3-b68f-908a0f1f328c/volumes"
Jul 29 15:31:19.900925 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:19.900886    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c73214-9mdxb"]
Jul 29 15:31:19.901333 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:19.901316    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6746c008-6c97-4473-909b-c5c9e4c90b7a" containerName="istio-proxy"
Jul 29 15:31:19.901333 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:19.901329    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6746c008-6c97-4473-909b-c5c9e4c90b7a" containerName="istio-proxy"
Jul 29 15:31:19.901421 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:19.901343    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6746c008-6c97-4473-909b-c5c9e4c90b7a" containerName="istio-proxy"
Jul 29 15:31:19.901421 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:19.901352    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6746c008-6c97-4473-909b-c5c9e4c90b7a" containerName="istio-proxy"
Jul 29 15:31:19.901421 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:19.901359    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b7550d2-d2f2-434b-a671-4ffd6d32f253" containerName="istio-proxy"
Jul 29 15:31:19.901421 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:19.901364    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b7550d2-d2f2-434b-a671-4ffd6d32f253" containerName="istio-proxy"
Jul 29 15:31:19.901421 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:19.901376    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0311700d-2984-4e8c-aedf-6df1efab1457" containerName="maas-api"
Jul 29 15:31:19.901421 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:19.901382    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0311700d-2984-4e8c-aedf-6df1efab1457" containerName="maas-api"
Jul 29 15:31:19.901421 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:19.901389    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55505629-2930-4ca3-b68f-908a0f1f328c" containerName="maas-api"
Jul 29 15:31:19.901421 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:19.901393    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="55505629-2930-4ca3-b68f-908a0f1f328c" containerName="maas-api"
Jul 29 15:31:19.901421 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:19.901400    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b7550d2-d2f2-434b-a671-4ffd6d32f253" containerName="istio-proxy"
Jul 29 15:31:19.901421 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:19.901407    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b7550d2-d2f2-434b-a671-4ffd6d32f253" containerName="istio-proxy"
Jul 29 15:31:19.901421 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:19.901414    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3776b8ec-0c4f-486c-b7e7-307e0c679c38" containerName="maas-api"
Jul 29 15:31:19.901421 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:19.901422    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3776b8ec-0c4f-486c-b7e7-307e0c679c38" containerName="maas-api"
Jul 29 15:31:19.901827 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:19.901485    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6746c008-6c97-4473-909b-c5c9e4c90b7a" containerName="istio-proxy"
Jul 29 15:31:19.901827 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:19.901496    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b7550d2-d2f2-434b-a671-4ffd6d32f253" containerName="istio-proxy"
Jul 29 15:31:19.901827 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:19.901505    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="55505629-2930-4ca3-b68f-908a0f1f328c" containerName="maas-api"
Jul 29 15:31:19.901827 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:19.901512    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3776b8ec-0c4f-486c-b7e7-307e0c679c38" containerName="maas-api"
Jul 29 15:31:19.901827 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:19.901519    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0311700d-2984-4e8c-aedf-6df1efab1457" containerName="maas-api"
Jul 29 15:31:19.901827 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:19.901523    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b7550d2-d2f2-434b-a671-4ffd6d32f253" containerName="istio-proxy"
Jul 29 15:31:19.904895 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:19.904873    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c73214-9mdxb"
Jul 29 15:31:19.911737 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:19.911710    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mmt6p\""
Jul 29 15:31:19.930347 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:19.930305    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c73214-9mdxb"]
Jul 29 15:31:20.023563 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:20.023525    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98271ec6-2862-46ae-831d-b67452a81126-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-c73214-9mdxb\" (UID: \"98271ec6-2862-46ae-831d-b67452a81126\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c73214-9mdxb"
Jul 29 15:31:20.124568 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:20.124531    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98271ec6-2862-46ae-831d-b67452a81126-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-c73214-9mdxb\" (UID: \"98271ec6-2862-46ae-831d-b67452a81126\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c73214-9mdxb"
Jul 29 15:31:20.125248 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:20.125160    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98271ec6-2862-46ae-831d-b67452a81126-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-c73214-9mdxb\" (UID: \"98271ec6-2862-46ae-831d-b67452a81126\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c73214-9mdxb"
Jul 29 15:31:20.214753 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:20.214716    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c73214-9mdxb"
Jul 29 15:31:20.382243 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:20.382219    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c73214-9mdxb"]
Jul 29 15:31:20.384623 ip-10-0-136-196 kubenswrapper[2563]: W0729 15:31:20.384593    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98271ec6_2862_46ae_831d_b67452a81126.slice/crio-8d0b217c70deab91bdf3a7ddf7d2fa80508c5d2ad2047b198e6e42381252896e WatchSource:0}: Error finding container 8d0b217c70deab91bdf3a7ddf7d2fa80508c5d2ad2047b198e6e42381252896e: Status 404 returned error can't find the container with id 8d0b217c70deab91bdf3a7ddf7d2fa80508c5d2ad2047b198e6e42381252896e
Jul 29 15:31:20.796312 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:20.795383    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c73214-9mdxb" event={"ID":"98271ec6-2862-46ae-831d-b67452a81126","Type":"ContainerStarted","Data":"bb4ae260cc695bdbad41053976e8eb75c15bfef052d405b2a4dbc429e8b7c59b"}
Jul 29 15:31:20.796312 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:20.795498    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c73214-9mdxb" event={"ID":"98271ec6-2862-46ae-831d-b67452a81126","Type":"ContainerStarted","Data":"8d0b217c70deab91bdf3a7ddf7d2fa80508c5d2ad2047b198e6e42381252896e"}
Jul 29 15:31:20.828586 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:20.828467    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c73214-9mdxb" podStartSLOduration=1.82845024 podStartE2EDuration="1.82845024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:31:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:31:20.826613591 +0000 UTC m=+3382.875071531" watchObservedRunningTime="2026-07-29 15:31:20.82845024 +0000 UTC m=+3382.876908179"
Jul 29 15:31:21.802017 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:21.801975    2563 generic.go:356] "Generic (PLEG): container finished" podID="98271ec6-2862-46ae-831d-b67452a81126" containerID="bb4ae260cc695bdbad41053976e8eb75c15bfef052d405b2a4dbc429e8b7c59b" exitCode=0
Jul 29 15:31:21.802456 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:21.802030    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c73214-9mdxb" event={"ID":"98271ec6-2862-46ae-831d-b67452a81126","Type":"ContainerDied","Data":"bb4ae260cc695bdbad41053976e8eb75c15bfef052d405b2a4dbc429e8b7c59b"}
Jul 29 15:31:22.944504 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:22.944474    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c73214-9mdxb"
Jul 29 15:31:23.053027 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:23.052986    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98271ec6-2862-46ae-831d-b67452a81126-maas-api-service-ca\") pod \"98271ec6-2862-46ae-831d-b67452a81126\" (UID: \"98271ec6-2862-46ae-831d-b67452a81126\") "
Jul 29 15:31:23.053400 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:23.053373    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98271ec6-2862-46ae-831d-b67452a81126-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "98271ec6-2862-46ae-831d-b67452a81126" (UID: "98271ec6-2862-46ae-831d-b67452a81126"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 15:31:23.154750 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:23.154652    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98271ec6-2862-46ae-831d-b67452a81126-maas-api-service-ca\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:31:23.813072 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:23.813038    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c73214-9mdxb" event={"ID":"98271ec6-2862-46ae-831d-b67452a81126","Type":"ContainerDied","Data":"8d0b217c70deab91bdf3a7ddf7d2fa80508c5d2ad2047b198e6e42381252896e"}
Jul 29 15:31:23.813072 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:23.813056    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c73214-9mdxb"
Jul 29 15:31:23.813307 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:23.813085    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d0b217c70deab91bdf3a7ddf7d2fa80508c5d2ad2047b198e6e42381252896e"
Jul 29 15:31:30.663204 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:30.663163    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c73214-6b499bc688-pc4tt"]
Jul 29 15:31:30.663690 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:30.663379    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c73214-6b499bc688-pc4tt" podUID="39f22377-bf8f-40ed-a965-d38999419d95" containerName="maas-api" containerID="cri-o://768951dc6cd5a0dc6902eda129d9d1a30b2a1d3202a83264903341f595632471" gracePeriod=30
Jul 29 15:31:30.851971 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:30.851935    2563 generic.go:356] "Generic (PLEG): container finished" podID="39f22377-bf8f-40ed-a965-d38999419d95" containerID="768951dc6cd5a0dc6902eda129d9d1a30b2a1d3202a83264903341f595632471" exitCode=0
Jul 29 15:31:30.852169 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:30.851983    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c73214-6b499bc688-pc4tt" event={"ID":"39f22377-bf8f-40ed-a965-d38999419d95","Type":"ContainerDied","Data":"768951dc6cd5a0dc6902eda129d9d1a30b2a1d3202a83264903341f595632471"}
Jul 29 15:31:30.929308 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:30.929236    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c73214-6b499bc688-pc4tt"
Jul 29 15:31:30.962739 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:30.962693    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c73214-9mdxb"]
Jul 29 15:31:30.966591 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:30.966563    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c73214-9mdxb"]
Jul 29 15:31:31.030437 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:31.030391    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39f22377-bf8f-40ed-a965-d38999419d95-openshift-service-ca\") pod \"39f22377-bf8f-40ed-a965-d38999419d95\" (UID: \"39f22377-bf8f-40ed-a965-d38999419d95\") "
Jul 29 15:31:31.030639 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:31.030446    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39f22377-bf8f-40ed-a965-d38999419d95-maas-api-tls\") pod \"39f22377-bf8f-40ed-a965-d38999419d95\" (UID: \"39f22377-bf8f-40ed-a965-d38999419d95\") "
Jul 29 15:31:31.030639 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:31.030495    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vp79s\" (UniqueName: \"kubernetes.io/projected/39f22377-bf8f-40ed-a965-d38999419d95-kube-api-access-vp79s\") pod \"39f22377-bf8f-40ed-a965-d38999419d95\" (UID: \"39f22377-bf8f-40ed-a965-d38999419d95\") "
Jul 29 15:31:31.030855 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:31.030827    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39f22377-bf8f-40ed-a965-d38999419d95-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "39f22377-bf8f-40ed-a965-d38999419d95" (UID: "39f22377-bf8f-40ed-a965-d38999419d95"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 15:31:31.032679 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:31.032650    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39f22377-bf8f-40ed-a965-d38999419d95-kube-api-access-vp79s" (OuterVolumeSpecName: "kube-api-access-vp79s") pod "39f22377-bf8f-40ed-a965-d38999419d95" (UID: "39f22377-bf8f-40ed-a965-d38999419d95"). InnerVolumeSpecName "kube-api-access-vp79s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 15:31:31.032679 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:31.032671    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39f22377-bf8f-40ed-a965-d38999419d95-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "39f22377-bf8f-40ed-a965-d38999419d95" (UID: "39f22377-bf8f-40ed-a965-d38999419d95"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 15:31:31.132278 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:31.132234    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39f22377-bf8f-40ed-a965-d38999419d95-maas-api-tls\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:31:31.132278 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:31.132275    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp79s\" (UniqueName: \"kubernetes.io/projected/39f22377-bf8f-40ed-a965-d38999419d95-kube-api-access-vp79s\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:31:31.132538 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:31.132290    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39f22377-bf8f-40ed-a965-d38999419d95-openshift-service-ca\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:31:31.858044 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:31.858014    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c73214-6b499bc688-pc4tt"
Jul 29 15:31:31.858496 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:31.858058    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c73214-6b499bc688-pc4tt" event={"ID":"39f22377-bf8f-40ed-a965-d38999419d95","Type":"ContainerDied","Data":"beba39f569abdaf9c149de8eb8b596edab1b73ab3d9a7939095535930a60b771"}
Jul 29 15:31:31.858496 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:31.858120    2563 scope.go:117] "RemoveContainer" containerID="768951dc6cd5a0dc6902eda129d9d1a30b2a1d3202a83264903341f595632471"
Jul 29 15:31:31.880965 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:31.880928    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c73214-6b499bc688-pc4tt"]
Jul 29 15:31:31.887493 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:31.887464    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c73214-6b499bc688-pc4tt"]
Jul 29 15:31:32.432117 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:32.432078    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39f22377-bf8f-40ed-a965-d38999419d95" path="/var/lib/kubelet/pods/39f22377-bf8f-40ed-a965-d38999419d95/volumes"
Jul 29 15:31:32.432494 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:32.432479    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98271ec6-2862-46ae-831d-b67452a81126" path="/var/lib/kubelet/pods/98271ec6-2862-46ae-831d-b67452a81126/volumes"
Jul 29 15:31:38.318711 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:38.318666    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-93e799-2c64s"]
Jul 29 15:31:38.319223 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:38.319072    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39f22377-bf8f-40ed-a965-d38999419d95" containerName="maas-api"
Jul 29 15:31:38.319223 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:38.319083    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="39f22377-bf8f-40ed-a965-d38999419d95" containerName="maas-api"
Jul 29 15:31:38.319223 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:38.319100    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98271ec6-2862-46ae-831d-b67452a81126" containerName="revoke-keys"
Jul 29 15:31:38.319223 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:38.319107    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="98271ec6-2862-46ae-831d-b67452a81126" containerName="revoke-keys"
Jul 29 15:31:38.319223 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:38.319213    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6746c008-6c97-4473-909b-c5c9e4c90b7a" containerName="istio-proxy"
Jul 29 15:31:38.319223 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:38.319226    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="98271ec6-2862-46ae-831d-b67452a81126" containerName="revoke-keys"
Jul 29 15:31:38.319421 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:38.319235    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="39f22377-bf8f-40ed-a965-d38999419d95" containerName="maas-api"
Jul 29 15:31:38.323798 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:38.323771    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-93e799-2c64s"
Jul 29 15:31:38.326184 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:38.326126    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mmt6p\""
Jul 29 15:31:38.332674 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:38.332645    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-93e799-2c64s"]
Jul 29 15:31:38.506013 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:38.505969    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a917e4de-a96f-4346-af59-768de116fda6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-93e799-2c64s\" (UID: \"a917e4de-a96f-4346-af59-768de116fda6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-93e799-2c64s"
Jul 29 15:31:38.607539 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:38.607443    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a917e4de-a96f-4346-af59-768de116fda6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-93e799-2c64s\" (UID: \"a917e4de-a96f-4346-af59-768de116fda6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-93e799-2c64s"
Jul 29 15:31:38.607996 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:38.607973    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a917e4de-a96f-4346-af59-768de116fda6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-93e799-2c64s\" (UID: \"a917e4de-a96f-4346-af59-768de116fda6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-93e799-2c64s"
Jul 29 15:31:38.635919 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:38.635884    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-93e799-2c64s"
Jul 29 15:31:38.798931 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:38.798895    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-93e799-2c64s"]
Jul 29 15:31:38.801092 ip-10-0-136-196 kubenswrapper[2563]: W0729 15:31:38.801062    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda917e4de_a96f_4346_af59_768de116fda6.slice/crio-3fe9f3474fd741d0b8e34bb9b67ada29f57ff52a578f1375ad193e9282e10735 WatchSource:0}: Error finding container 3fe9f3474fd741d0b8e34bb9b67ada29f57ff52a578f1375ad193e9282e10735: Status 404 returned error can't find the container with id 3fe9f3474fd741d0b8e34bb9b67ada29f57ff52a578f1375ad193e9282e10735
Jul 29 15:31:38.900761 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:38.900657    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-93e799-2c64s" event={"ID":"a917e4de-a96f-4346-af59-768de116fda6","Type":"ContainerStarted","Data":"3fe9f3474fd741d0b8e34bb9b67ada29f57ff52a578f1375ad193e9282e10735"}
Jul 29 15:31:39.907910 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:39.907804    2563 generic.go:356] "Generic (PLEG): container finished" podID="a917e4de-a96f-4346-af59-768de116fda6" containerID="adbcb508b3169c499fc534046628fb008b7eb7e25b40cbfd69017103ca8479fe" exitCode=0
Jul 29 15:31:39.907910 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:39.907893    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-93e799-2c64s" event={"ID":"a917e4de-a96f-4346-af59-768de116fda6","Type":"ContainerDied","Data":"adbcb508b3169c499fc534046628fb008b7eb7e25b40cbfd69017103ca8479fe"}
Jul 29 15:31:41.051702 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:41.051673    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-93e799-2c64s"
Jul 29 15:31:41.129217 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:41.129176    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a917e4de-a96f-4346-af59-768de116fda6-maas-api-service-ca\") pod \"a917e4de-a96f-4346-af59-768de116fda6\" (UID: \"a917e4de-a96f-4346-af59-768de116fda6\") "
Jul 29 15:31:41.129556 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:41.129533    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a917e4de-a96f-4346-af59-768de116fda6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a917e4de-a96f-4346-af59-768de116fda6" (UID: "a917e4de-a96f-4346-af59-768de116fda6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 15:31:41.229848 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:41.229805    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a917e4de-a96f-4346-af59-768de116fda6-maas-api-service-ca\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:31:41.919359 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:41.919329    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-93e799-2c64s"
Jul 29 15:31:41.919535 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:41.919327    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-93e799-2c64s" event={"ID":"a917e4de-a96f-4346-af59-768de116fda6","Type":"ContainerDied","Data":"3fe9f3474fd741d0b8e34bb9b67ada29f57ff52a578f1375ad193e9282e10735"}
Jul 29 15:31:41.919535 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:41.919487    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fe9f3474fd741d0b8e34bb9b67ada29f57ff52a578f1375ad193e9282e10735"
Jul 29 15:31:48.874242 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:48.874194    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-93e799-5bdd9c84bc-r8l9m"]
Jul 29 15:31:48.874749 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:48.874421    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-93e799-5bdd9c84bc-r8l9m" podUID="b09574c0-d7c9-4cef-99f9-ca1cadab0af6" containerName="maas-api" containerID="cri-o://23a5214f7551f5ae6afaa89073fb842ff21e4d24a4cdfccb7673dae51bb18e71" gracePeriod=30
Jul 29 15:31:49.132944 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:49.132833    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-93e799-2c64s"]
Jul 29 15:31:49.135596 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:49.135564    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-93e799-2c64s"]
Jul 29 15:31:49.137126 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:49.137108    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-93e799-5bdd9c84bc-r8l9m"
Jul 29 15:31:49.207434 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:49.207392    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b09574c0-d7c9-4cef-99f9-ca1cadab0af6-openshift-service-ca\") pod \"b09574c0-d7c9-4cef-99f9-ca1cadab0af6\" (UID: \"b09574c0-d7c9-4cef-99f9-ca1cadab0af6\") "
Jul 29 15:31:49.207640 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:49.207446    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddlxh\" (UniqueName: \"kubernetes.io/projected/b09574c0-d7c9-4cef-99f9-ca1cadab0af6-kube-api-access-ddlxh\") pod \"b09574c0-d7c9-4cef-99f9-ca1cadab0af6\" (UID: \"b09574c0-d7c9-4cef-99f9-ca1cadab0af6\") "
Jul 29 15:31:49.207640 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:49.207494    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b09574c0-d7c9-4cef-99f9-ca1cadab0af6-maas-api-tls\") pod \"b09574c0-d7c9-4cef-99f9-ca1cadab0af6\" (UID: \"b09574c0-d7c9-4cef-99f9-ca1cadab0af6\") "
Jul 29 15:31:49.207771 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:49.207749    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b09574c0-d7c9-4cef-99f9-ca1cadab0af6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b09574c0-d7c9-4cef-99f9-ca1cadab0af6" (UID: "b09574c0-d7c9-4cef-99f9-ca1cadab0af6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 15:31:49.209817 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:49.209781    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b09574c0-d7c9-4cef-99f9-ca1cadab0af6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b09574c0-d7c9-4cef-99f9-ca1cadab0af6" (UID: "b09574c0-d7c9-4cef-99f9-ca1cadab0af6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 15:31:49.209958 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:49.209821    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b09574c0-d7c9-4cef-99f9-ca1cadab0af6-kube-api-access-ddlxh" (OuterVolumeSpecName: "kube-api-access-ddlxh") pod "b09574c0-d7c9-4cef-99f9-ca1cadab0af6" (UID: "b09574c0-d7c9-4cef-99f9-ca1cadab0af6"). InnerVolumeSpecName "kube-api-access-ddlxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 15:31:49.308230 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:49.308195    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b09574c0-d7c9-4cef-99f9-ca1cadab0af6-openshift-service-ca\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:31:49.308230 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:49.308228    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddlxh\" (UniqueName: \"kubernetes.io/projected/b09574c0-d7c9-4cef-99f9-ca1cadab0af6-kube-api-access-ddlxh\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:31:49.308454 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:49.308242    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b09574c0-d7c9-4cef-99f9-ca1cadab0af6-maas-api-tls\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:31:49.965204 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:49.965168    2563 generic.go:356] "Generic (PLEG): container finished" podID="b09574c0-d7c9-4cef-99f9-ca1cadab0af6" containerID="23a5214f7551f5ae6afaa89073fb842ff21e4d24a4cdfccb7673dae51bb18e71" exitCode=0
Jul 29 15:31:49.965684 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:49.965219    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-93e799-5bdd9c84bc-r8l9m"
Jul 29 15:31:49.965684 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:49.965231    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-93e799-5bdd9c84bc-r8l9m" event={"ID":"b09574c0-d7c9-4cef-99f9-ca1cadab0af6","Type":"ContainerDied","Data":"23a5214f7551f5ae6afaa89073fb842ff21e4d24a4cdfccb7673dae51bb18e71"}
Jul 29 15:31:49.965684 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:49.965285    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-93e799-5bdd9c84bc-r8l9m" event={"ID":"b09574c0-d7c9-4cef-99f9-ca1cadab0af6","Type":"ContainerDied","Data":"62b12eae6b77f141cc004ed2bff231cac468024008f63f06ffc957ef45acf7e6"}
Jul 29 15:31:49.965684 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:49.965318    2563 scope.go:117] "RemoveContainer" containerID="23a5214f7551f5ae6afaa89073fb842ff21e4d24a4cdfccb7673dae51bb18e71"
Jul 29 15:31:49.975213 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:49.975184    2563 scope.go:117] "RemoveContainer" containerID="23a5214f7551f5ae6afaa89073fb842ff21e4d24a4cdfccb7673dae51bb18e71"
Jul 29 15:31:49.975549 ip-10-0-136-196 kubenswrapper[2563]: E0729 15:31:49.975530    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23a5214f7551f5ae6afaa89073fb842ff21e4d24a4cdfccb7673dae51bb18e71\": container with ID starting with 23a5214f7551f5ae6afaa89073fb842ff21e4d24a4cdfccb7673dae51bb18e71 not found: ID does not exist" containerID="23a5214f7551f5ae6afaa89073fb842ff21e4d24a4cdfccb7673dae51bb18e71"
Jul 29 15:31:49.975601 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:49.975560    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23a5214f7551f5ae6afaa89073fb842ff21e4d24a4cdfccb7673dae51bb18e71"} err="failed to get container status \"23a5214f7551f5ae6afaa89073fb842ff21e4d24a4cdfccb7673dae51bb18e71\": rpc error: code = NotFound desc = could not find container \"23a5214f7551f5ae6afaa89073fb842ff21e4d24a4cdfccb7673dae51bb18e71\": container with ID starting with 23a5214f7551f5ae6afaa89073fb842ff21e4d24a4cdfccb7673dae51bb18e71 not found: ID does not exist"
Jul 29 15:31:49.986525 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:49.986490    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-93e799-5bdd9c84bc-r8l9m"]
Jul 29 15:31:49.990081 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:49.990051    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-93e799-5bdd9c84bc-r8l9m"]
Jul 29 15:31:50.432984 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:50.432896    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a917e4de-a96f-4346-af59-768de116fda6" path="/var/lib/kubelet/pods/a917e4de-a96f-4346-af59-768de116fda6/volumes"
Jul 29 15:31:50.433366 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:50.433350    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b09574c0-d7c9-4cef-99f9-ca1cadab0af6" path="/var/lib/kubelet/pods/b09574c0-d7c9-4cef-99f9-ca1cadab0af6/volumes"
Jul 29 15:31:55.785764 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:55.785717    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"]
Jul 29 15:31:55.786293 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:55.785940    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7" podUID="a406c1c9-fcaf-4d16-aa78-0383a59e3260" containerName="istio-proxy" containerID="cri-o://dfec5e727ddb4d2b77e7c2fac4fa0fdbc7430a0c5e3446f033e2d21a01588182" gracePeriod=30
Jul 29 15:31:56.415412 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:56.415372    2563 patch_prober.go:28] interesting pod/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 29 15:31:56.415595 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:31:56.415450    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7" podUID="a406c1c9-fcaf-4d16-aa78-0383a59e3260" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 15:32:01.024324 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:01.024285    2563 generic.go:356] "Generic (PLEG): container finished" podID="a406c1c9-fcaf-4d16-aa78-0383a59e3260" containerID="dfec5e727ddb4d2b77e7c2fac4fa0fdbc7430a0c5e3446f033e2d21a01588182" exitCode=0
Jul 29 15:32:01.024324 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:01.024329    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7" event={"ID":"a406c1c9-fcaf-4d16-aa78-0383a59e3260","Type":"ContainerDied","Data":"dfec5e727ddb4d2b77e7c2fac4fa0fdbc7430a0c5e3446f033e2d21a01588182"}
Jul 29 15:32:01.024827 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:01.024378    2563 scope.go:117] "RemoveContainer" containerID="c94826643f6a1bcc1b1e86cfd07a82188eb216571fc6a0769efe37e0afba8292"
Jul 29 15:32:01.059525 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:01.059453    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"
Jul 29 15:32:01.115561 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:01.115524    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a406c1c9-fcaf-4d16-aa78-0383a59e3260-istiod-ca-cert\") pod \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\" (UID: \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\") "
Jul 29 15:32:01.115759 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:01.115583    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a406c1c9-fcaf-4d16-aa78-0383a59e3260-istio-token\") pod \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\" (UID: \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\") "
Jul 29 15:32:01.115759 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:01.115602    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a406c1c9-fcaf-4d16-aa78-0383a59e3260-workload-certs\") pod \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\" (UID: \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\") "
Jul 29 15:32:01.115759 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:01.115741    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a406c1c9-fcaf-4d16-aa78-0383a59e3260-istio-data\") pod \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\" (UID: \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\") "
Jul 29 15:32:01.115927 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:01.115795    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a406c1c9-fcaf-4d16-aa78-0383a59e3260-workload-socket\") pod \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\" (UID: \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\") "
Jul 29 15:32:01.115927 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:01.115874    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vg4r8\" (UniqueName: \"kubernetes.io/projected/a406c1c9-fcaf-4d16-aa78-0383a59e3260-kube-api-access-vg4r8\") pod \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\" (UID: \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\") "
Jul 29 15:32:01.115927 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:01.115901    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a406c1c9-fcaf-4d16-aa78-0383a59e3260-istio-envoy\") pod \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\" (UID: \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\") "
Jul 29 15:32:01.115927 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:01.115894    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a406c1c9-fcaf-4d16-aa78-0383a59e3260-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a406c1c9-fcaf-4d16-aa78-0383a59e3260" (UID: "a406c1c9-fcaf-4d16-aa78-0383a59e3260"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:32:01.116116 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:01.115936    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a406c1c9-fcaf-4d16-aa78-0383a59e3260-credential-socket\") pod \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\" (UID: \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\") "
Jul 29 15:32:01.116116 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:01.115975    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a406c1c9-fcaf-4d16-aa78-0383a59e3260-istio-podinfo\") pod \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\" (UID: \"a406c1c9-fcaf-4d16-aa78-0383a59e3260\") "
Jul 29 15:32:01.116116 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:01.116094    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a406c1c9-fcaf-4d16-aa78-0383a59e3260-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a406c1c9-fcaf-4d16-aa78-0383a59e3260" (UID: "a406c1c9-fcaf-4d16-aa78-0383a59e3260"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:32:01.116323 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:01.116111    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a406c1c9-fcaf-4d16-aa78-0383a59e3260-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a406c1c9-fcaf-4d16-aa78-0383a59e3260" (UID: "a406c1c9-fcaf-4d16-aa78-0383a59e3260"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 15:32:01.116323 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:01.116278    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a406c1c9-fcaf-4d16-aa78-0383a59e3260-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a406c1c9-fcaf-4d16-aa78-0383a59e3260" (UID: "a406c1c9-fcaf-4d16-aa78-0383a59e3260"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:32:01.116402 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:01.116353    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a406c1c9-fcaf-4d16-aa78-0383a59e3260-workload-socket\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:32:01.116402 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:01.116373    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a406c1c9-fcaf-4d16-aa78-0383a59e3260-credential-socket\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:32:01.116402 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:01.116387    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a406c1c9-fcaf-4d16-aa78-0383a59e3260-istiod-ca-cert\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:32:01.116402 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:01.116398    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a406c1c9-fcaf-4d16-aa78-0383a59e3260-workload-certs\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:32:01.116753 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:01.116732    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a406c1c9-fcaf-4d16-aa78-0383a59e3260-istio-data" (OuterVolumeSpecName: "istio-data") pod "a406c1c9-fcaf-4d16-aa78-0383a59e3260" (UID: "a406c1c9-fcaf-4d16-aa78-0383a59e3260"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:32:01.117817 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:01.117790    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a406c1c9-fcaf-4d16-aa78-0383a59e3260-istio-token" (OuterVolumeSpecName: "istio-token") pod "a406c1c9-fcaf-4d16-aa78-0383a59e3260" (UID: "a406c1c9-fcaf-4d16-aa78-0383a59e3260"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 15:32:01.118118 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:01.118100    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a406c1c9-fcaf-4d16-aa78-0383a59e3260-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a406c1c9-fcaf-4d16-aa78-0383a59e3260" (UID: "a406c1c9-fcaf-4d16-aa78-0383a59e3260"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 15:32:01.118465 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:01.118444    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a406c1c9-fcaf-4d16-aa78-0383a59e3260-kube-api-access-vg4r8" (OuterVolumeSpecName: "kube-api-access-vg4r8") pod "a406c1c9-fcaf-4d16-aa78-0383a59e3260" (UID: "a406c1c9-fcaf-4d16-aa78-0383a59e3260"). InnerVolumeSpecName "kube-api-access-vg4r8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 15:32:01.118537 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:01.118488    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a406c1c9-fcaf-4d16-aa78-0383a59e3260-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a406c1c9-fcaf-4d16-aa78-0383a59e3260" (UID: "a406c1c9-fcaf-4d16-aa78-0383a59e3260"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 15:32:01.217405 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:01.217359    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a406c1c9-fcaf-4d16-aa78-0383a59e3260-istio-data\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:32:01.217405 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:01.217400    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg4r8\" (UniqueName: \"kubernetes.io/projected/a406c1c9-fcaf-4d16-aa78-0383a59e3260-kube-api-access-vg4r8\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:32:01.217405 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:01.217414    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a406c1c9-fcaf-4d16-aa78-0383a59e3260-istio-envoy\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:32:01.217661 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:01.217426    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a406c1c9-fcaf-4d16-aa78-0383a59e3260-istio-podinfo\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:32:01.217661 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:01.217437    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a406c1c9-fcaf-4d16-aa78-0383a59e3260-istio-token\") on node \"ip-10-0-136-196.ec2.internal\" DevicePath \"\""
Jul 29 15:32:02.033222 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:02.033179    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7" event={"ID":"a406c1c9-fcaf-4d16-aa78-0383a59e3260","Type":"ContainerDied","Data":"08405a7cd44d077820bc59d11e9463f3ad7ea943d9e210748ce8a29aba73181c"}
Jul 29 15:32:02.033710 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:02.033260    2563 scope.go:117] "RemoveContainer" containerID="dfec5e727ddb4d2b77e7c2fac4fa0fdbc7430a0c5e3446f033e2d21a01588182"
Jul 29 15:32:02.033710 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:02.033274    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"
Jul 29 15:32:02.057662 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:02.057626    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"]
Jul 29 15:32:02.061472 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:02.061439    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-93e799-openshift-default-7975b565bd-pzxh7"]
Jul 29 15:32:02.172363 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:02.172297    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-794c9d7b98-w5xq8_d0858163-11e4-4dec-bd96-ca7ceb51e90e/manager/0.log"
Jul 29 15:32:02.432390 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:02.432301    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a406c1c9-fcaf-4d16-aa78-0383a59e3260" path="/var/lib/kubelet/pods/a406c1c9-fcaf-4d16-aa78-0383a59e3260/volumes"
Jul 29 15:32:02.781967 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:02.781940    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-65f9f6cd8c-m78x8_be465c39-a202-4767-8e93-907564622462/copy-manifests/0.log"
Jul 29 15:32:02.789618 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:02.789589    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-65f9f6cd8c-m78x8_be465c39-a202-4767-8e93-907564622462/manager/0.log"
Jul 29 15:32:03.148033 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:03.147945    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-55fcb658db-kl2k7_2cc2924d-9b81-44dc-bf27-6bc6a7472b06/maas-api/0.log"
Jul 29 15:32:03.610641 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:03.610615    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-kq4j9_c3cf1288-f7de-424e-b18c-f3e8c882d07c/postgres/0.log"
Jul 29 15:32:05.164908 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:05.164872    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-bf7t2_93d6e8f8-836e-4538-9326-c81fd86c1fe0/manager/0.log"
Jul 29 15:32:05.387570 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:05.387535    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-t8vp6_80ed5604-3b78-4658-8263-63d431ed86d5/registry-server/0.log"
Jul 29 15:32:05.751894 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:05.751855    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-8s4rc_7445057f-7eb6-4532-ae7b-e9422ce7cb7f/manager/0.log"
Jul 29 15:32:06.929674 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:06.929637    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-746f69dff9-2bdj7_1a5f8182-95a6-4f4b-b40b-ea003c7be4dd/router/0.log"
Jul 29 15:32:07.274004 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:07.273956    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-x8mv2_1f545ede-4a2b-4e4d-996a-ec9628316097/main/0.log"
Jul 29 15:32:07.608025 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:07.607946    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bpms_5b39f59c-2d08-479b-b89f-d9f47a342d5f/main/0.log"
Jul 29 15:32:07.726116 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:07.726079    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-47crv_93de9b1a-dbb7-4255-b4b3-8e96ca7e2b6c/main/0.log"
Jul 29 15:32:16.795127 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:16.795091    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vg8f5_95701def-c5e7-4309-a07b-6a7c05316dd6/global-pull-secret-syncer/0.log"
Jul 29 15:32:16.906723 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:16.906683    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-k2jpl_9a100b51-c157-4a63-a6c5-9d92a3fee8a7/konnectivity-agent/0.log"
Jul 29 15:32:16.948957 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:16.948908    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-196.ec2.internal_c88b7e35b1a8b1d19afb95df94d0ce5f/haproxy/0.log"
Jul 29 15:32:22.298494 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:22.298462    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-bf7t2_93d6e8f8-836e-4538-9326-c81fd86c1fe0/manager/0.log"
Jul 29 15:32:22.365794 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:22.365758    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-t8vp6_80ed5604-3b78-4658-8263-63d431ed86d5/registry-server/0.log"
Jul 29 15:32:22.624470 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:22.624385    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-8s4rc_7445057f-7eb6-4532-ae7b-e9422ce7cb7f/manager/0.log"
Jul 29 15:32:24.353193 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:24.353122    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-bxjbs_f2037177-a695-47b3-bda0-0669bcfe408e/cluster-monitoring-operator/0.log"
Jul 29 15:32:24.505609 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:24.505572    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8zfjf_8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85/node-exporter/0.log"
Jul 29 15:32:24.538830 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:24.538793    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8zfjf_8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85/kube-rbac-proxy/0.log"
Jul 29 15:32:24.566545 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:24.566519    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8zfjf_8c0a2c1f-1b0d-4f90-bb78-dd2ee6dede85/init-textfile/0.log"
Jul 29 15:32:26.379249 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:26.379220    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-qlc8b_b6a44e5c-e648-4bb8-a393-7cfb4c22d761/networking-console-plugin/0.log"
Jul 29 15:32:26.764153 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:26.764097    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mh6mr/perf-node-gather-daemonset-9djxp"]
Jul 29 15:32:26.764484 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:26.764471    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a406c1c9-fcaf-4d16-aa78-0383a59e3260" containerName="istio-proxy"
Jul 29 15:32:26.764532 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:26.764486    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a406c1c9-fcaf-4d16-aa78-0383a59e3260" containerName="istio-proxy"
Jul 29 15:32:26.764532 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:26.764499    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a406c1c9-fcaf-4d16-aa78-0383a59e3260" containerName="istio-proxy"
Jul 29 15:32:26.764532 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:26.764505    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a406c1c9-fcaf-4d16-aa78-0383a59e3260" containerName="istio-proxy"
Jul 29 15:32:26.764532 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:26.764523    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a917e4de-a96f-4346-af59-768de116fda6" containerName="revoke-keys"
Jul 29 15:32:26.764532 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:26.764528    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a917e4de-a96f-4346-af59-768de116fda6" containerName="revoke-keys"
Jul 29 15:32:26.764687 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:26.764538    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b09574c0-d7c9-4cef-99f9-ca1cadab0af6" containerName="maas-api"
Jul 29 15:32:26.764687 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:26.764544    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b09574c0-d7c9-4cef-99f9-ca1cadab0af6" containerName="maas-api"
Jul 29 15:32:26.764687 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:26.764593    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a406c1c9-fcaf-4d16-aa78-0383a59e3260" containerName="istio-proxy"
Jul 29 15:32:26.764687 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:26.764603    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b09574c0-d7c9-4cef-99f9-ca1cadab0af6" containerName="maas-api"
Jul 29 15:32:26.764687 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:26.764609    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a917e4de-a96f-4346-af59-768de116fda6" containerName="revoke-keys"
Jul 29 15:32:26.764687 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:26.764615    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a406c1c9-fcaf-4d16-aa78-0383a59e3260" containerName="istio-proxy"
Jul 29 15:32:26.767696 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:26.767679    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mh6mr/perf-node-gather-daemonset-9djxp"
Jul 29 15:32:26.771473 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:26.771343    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mh6mr\"/\"kube-root-ca.crt\""
Jul 29 15:32:26.772257 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:26.772232    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mh6mr\"/\"openshift-service-ca.crt\""
Jul 29 15:32:26.772257 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:26.772252    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mh6mr\"/\"default-dockercfg-t5t7t\""
Jul 29 15:32:26.777371 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:26.777344    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mh6mr/perf-node-gather-daemonset-9djxp"]
Jul 29 15:32:26.853412 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:26.853375    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f95eb06-6cde-4937-89e1-24e7c5e67e79-sys\") pod \"perf-node-gather-daemonset-9djxp\" (UID: \"0f95eb06-6cde-4937-89e1-24e7c5e67e79\") " pod="openshift-must-gather-mh6mr/perf-node-gather-daemonset-9djxp"
Jul 29 15:32:26.853615 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:26.853447    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f95eb06-6cde-4937-89e1-24e7c5e67e79-lib-modules\") pod \"perf-node-gather-daemonset-9djxp\" (UID: \"0f95eb06-6cde-4937-89e1-24e7c5e67e79\") " pod="openshift-must-gather-mh6mr/perf-node-gather-daemonset-9djxp"
Jul 29 15:32:26.853615 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:26.853476    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmmn4\" (UniqueName: \"kubernetes.io/projected/0f95eb06-6cde-4937-89e1-24e7c5e67e79-kube-api-access-lmmn4\") pod \"perf-node-gather-daemonset-9djxp\" (UID: \"0f95eb06-6cde-4937-89e1-24e7c5e67e79\") " pod="openshift-must-gather-mh6mr/perf-node-gather-daemonset-9djxp"
Jul 29 15:32:26.853615 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:26.853508    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0f95eb06-6cde-4937-89e1-24e7c5e67e79-proc\") pod \"perf-node-gather-daemonset-9djxp\" (UID: \"0f95eb06-6cde-4937-89e1-24e7c5e67e79\") " pod="openshift-must-gather-mh6mr/perf-node-gather-daemonset-9djxp"
Jul 29 15:32:26.853615 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:26.853558    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0f95eb06-6cde-4937-89e1-24e7c5e67e79-podres\") pod \"perf-node-gather-daemonset-9djxp\" (UID: \"0f95eb06-6cde-4937-89e1-24e7c5e67e79\") " pod="openshift-must-gather-mh6mr/perf-node-gather-daemonset-9djxp"
Jul 29 15:32:26.953989 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:26.953952    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0f95eb06-6cde-4937-89e1-24e7c5e67e79-proc\") pod \"perf-node-gather-daemonset-9djxp\" (UID: \"0f95eb06-6cde-4937-89e1-24e7c5e67e79\") " pod="openshift-must-gather-mh6mr/perf-node-gather-daemonset-9djxp"
Jul 29 15:32:26.953989 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:26.953996    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0f95eb06-6cde-4937-89e1-24e7c5e67e79-podres\") pod \"perf-node-gather-daemonset-9djxp\" (UID: \"0f95eb06-6cde-4937-89e1-24e7c5e67e79\") " pod="openshift-must-gather-mh6mr/perf-node-gather-daemonset-9djxp"
Jul 29 15:32:26.954277 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:26.954034    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f95eb06-6cde-4937-89e1-24e7c5e67e79-sys\") pod \"perf-node-gather-daemonset-9djxp\" (UID: \"0f95eb06-6cde-4937-89e1-24e7c5e67e79\") " pod="openshift-must-gather-mh6mr/perf-node-gather-daemonset-9djxp"
Jul 29 15:32:26.954277 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:26.954058    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0f95eb06-6cde-4937-89e1-24e7c5e67e79-proc\") pod \"perf-node-gather-daemonset-9djxp\" (UID: \"0f95eb06-6cde-4937-89e1-24e7c5e67e79\") " pod="openshift-must-gather-mh6mr/perf-node-gather-daemonset-9djxp"
Jul 29 15:32:26.954277 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:26.954067    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f95eb06-6cde-4937-89e1-24e7c5e67e79-lib-modules\") pod \"perf-node-gather-daemonset-9djxp\" (UID: \"0f95eb06-6cde-4937-89e1-24e7c5e67e79\") " pod="openshift-must-gather-mh6mr/perf-node-gather-daemonset-9djxp"
Jul 29 15:32:26.954277 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:26.954114    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmmn4\" (UniqueName: \"kubernetes.io/projected/0f95eb06-6cde-4937-89e1-24e7c5e67e79-kube-api-access-lmmn4\") pod \"perf-node-gather-daemonset-9djxp\" (UID: \"0f95eb06-6cde-4937-89e1-24e7c5e67e79\") " pod="openshift-must-gather-mh6mr/perf-node-gather-daemonset-9djxp"
Jul 29 15:32:26.954277 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:26.954167    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f95eb06-6cde-4937-89e1-24e7c5e67e79-sys\") pod \"perf-node-gather-daemonset-9djxp\" (UID: \"0f95eb06-6cde-4937-89e1-24e7c5e67e79\") " pod="openshift-must-gather-mh6mr/perf-node-gather-daemonset-9djxp"
Jul 29 15:32:26.954277 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:26.954185    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0f95eb06-6cde-4937-89e1-24e7c5e67e79-podres\") pod \"perf-node-gather-daemonset-9djxp\" (UID: \"0f95eb06-6cde-4937-89e1-24e7c5e67e79\") " pod="openshift-must-gather-mh6mr/perf-node-gather-daemonset-9djxp"
Jul 29 15:32:26.954277 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:26.954197    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f95eb06-6cde-4937-89e1-24e7c5e67e79-lib-modules\") pod \"perf-node-gather-daemonset-9djxp\" (UID: \"0f95eb06-6cde-4937-89e1-24e7c5e67e79\") " pod="openshift-must-gather-mh6mr/perf-node-gather-daemonset-9djxp"
Jul 29 15:32:26.962412 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:26.962378    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmmn4\" (UniqueName: \"kubernetes.io/projected/0f95eb06-6cde-4937-89e1-24e7c5e67e79-kube-api-access-lmmn4\") pod \"perf-node-gather-daemonset-9djxp\" (UID: \"0f95eb06-6cde-4937-89e1-24e7c5e67e79\") " pod="openshift-must-gather-mh6mr/perf-node-gather-daemonset-9djxp"
Jul 29 15:32:26.993994 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:26.993962    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xnm95_23d79588-937d-4c6d-8760-a081024f608e/console-operator/1.log"
Jul 29 15:32:26.998391 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:26.998346    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xnm95_23d79588-937d-4c6d-8760-a081024f608e/console-operator/2.log"
Jul 29 15:32:27.078647 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:27.078552    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mh6mr/perf-node-gather-daemonset-9djxp"
Jul 29 15:32:27.219302 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:27.219268    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mh6mr/perf-node-gather-daemonset-9djxp"]
Jul 29 15:32:27.221408 ip-10-0-136-196 kubenswrapper[2563]: W0729 15:32:27.221380    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0f95eb06_6cde_4937_89e1_24e7c5e67e79.slice/crio-23b00391d45c859dd409fc0a58687eb620cf198baf68253ad5e694bc8f9935e9 WatchSource:0}: Error finding container 23b00391d45c859dd409fc0a58687eb620cf198baf68253ad5e694bc8f9935e9: Status 404 returned error can't find the container with id 23b00391d45c859dd409fc0a58687eb620cf198baf68253ad5e694bc8f9935e9
Jul 29 15:32:28.110935 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:28.110901    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-g9dbl_c1eb675a-8bbd-48ef-a526-506a7fd20ee7/volume-data-source-validator/0.log"
Jul 29 15:32:28.163824 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:28.163785    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mh6mr/perf-node-gather-daemonset-9djxp" event={"ID":"0f95eb06-6cde-4937-89e1-24e7c5e67e79","Type":"ContainerStarted","Data":"85ebd56468fd10c6126911952674121b2846c170a315d80598534415564cef67"}
Jul 29 15:32:28.164029 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:28.163866    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mh6mr/perf-node-gather-daemonset-9djxp" event={"ID":"0f95eb06-6cde-4937-89e1-24e7c5e67e79","Type":"ContainerStarted","Data":"23b00391d45c859dd409fc0a58687eb620cf198baf68253ad5e694bc8f9935e9"}
Jul 29 15:32:28.164029 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:28.163888    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mh6mr/perf-node-gather-daemonset-9djxp"
Jul 29 15:32:28.180121 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:28.180065    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mh6mr/perf-node-gather-daemonset-9djxp" podStartSLOduration=2.18004699 podStartE2EDuration="2.18004699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 15:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:32:28.178445083 +0000 UTC m=+3450.226903022" watchObservedRunningTime="2026-07-29 15:32:28.18004699 +0000 UTC m=+3450.228504928"
Jul 29 15:32:29.020829 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:29.020802    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4m4xj_4dbfdc61-952e-49ee-a2b2-a484d295b1e3/dns/0.log"
Jul 29 15:32:29.041689 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:29.041651    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4m4xj_4dbfdc61-952e-49ee-a2b2-a484d295b1e3/kube-rbac-proxy/0.log"
Jul 29 15:32:29.248063 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:29.248029    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n7qwt_8a010eab-ffa4-4f5a-a535-ecae3e55cf16/dns-node-resolver/0.log"
Jul 29 15:32:29.772837 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:29.772803    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d44hd_75a67018-8010-4684-ac1c-4a12c7f30bea/node-ca/0.log"
Jul 29 15:32:30.997287 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:30.997258    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-746f69dff9-2bdj7_1a5f8182-95a6-4f4b-b40b-ea003c7be4dd/router/0.log"
Jul 29 15:32:31.786883 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:31.786851    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qcjfr_031b9fd3-1851-451a-b5e4-7f667c641263/serve-healthcheck-canary/0.log"
Jul 29 15:32:32.227118 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:32.227084    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-nh8pd_15611a65-9454-4396-9b2f-fc9599672fc4/insights-operator/0.log"
Jul 29 15:32:32.228003 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:32.227981    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-nh8pd_15611a65-9454-4396-9b2f-fc9599672fc4/insights-operator/1.log"
Jul 29 15:32:32.246830 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:32.246801    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mvd8l_d86b4f92-11dc-402c-80cd-53bbb1981778/kube-rbac-proxy/0.log"
Jul 29 15:32:32.265823 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:32.265792    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mvd8l_d86b4f92-11dc-402c-80cd-53bbb1981778/exporter/0.log"
Jul 29 15:32:32.287803 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:32.287775    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mvd8l_d86b4f92-11dc-402c-80cd-53bbb1981778/extractor/0.log"
Jul 29 15:32:34.178621 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:34.178594    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mh6mr/perf-node-gather-daemonset-9djxp"
Jul 29 15:32:34.696736 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:34.696696    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-794c9d7b98-w5xq8_d0858163-11e4-4dec-bd96-ca7ceb51e90e/manager/0.log"
Jul 29 15:32:34.990639 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:34.990608    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-65f9f6cd8c-m78x8_be465c39-a202-4767-8e93-907564622462/manager/0.log"
Jul 29 15:32:35.014049 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:35.014025    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-65f9f6cd8c-m78x8_be465c39-a202-4767-8e93-907564622462/copy-manifests/0.log"
Jul 29 15:32:36.410451 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:36.410420    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-746f69dff9-2bdj7_1a5f8182-95a6-4f4b-b40b-ea003c7be4dd/router/0.log"
Jul 29 15:32:36.413505 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:36.413481    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-856778fb54-t2kfl_2f06b59e-f9ff-4761-b638-442843e96949/manager/0.log"
Jul 29 15:32:38.007683 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:38.007655    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-746f69dff9-2bdj7_1a5f8182-95a6-4f4b-b40b-ea003c7be4dd/router/0.log"
Jul 29 15:32:41.265942 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:41.265914    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-hqn29_14d576fd-12e6-4370-9499-2112ba0c9eb7/migrator/0.log"
Jul 29 15:32:41.283947 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:41.283916    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-hqn29_14d576fd-12e6-4370-9499-2112ba0c9eb7/graceful-termination/0.log"
Jul 29 15:32:41.613638 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:41.613550    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-fkjfz_7504b97a-38e2-44ff-822a-a7f1e98b8933/kube-storage-version-migrator-operator/1.log"
Jul 29 15:32:41.614501 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:41.614479    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-fkjfz_7504b97a-38e2-44ff-822a-a7f1e98b8933/kube-storage-version-migrator-operator/0.log"
Jul 29 15:32:42.687387 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:42.687352    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nwj5f_e666f634-f60c-47fb-952c-7e18a781609b/kube-multus-additional-cni-plugins/0.log"
Jul 29 15:32:42.710354 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:42.710326    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nwj5f_e666f634-f60c-47fb-952c-7e18a781609b/egress-router-binary-copy/0.log"
Jul 29 15:32:42.730867 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:42.730841    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nwj5f_e666f634-f60c-47fb-952c-7e18a781609b/cni-plugins/0.log"
Jul 29 15:32:42.751866 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:42.751842    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nwj5f_e666f634-f60c-47fb-952c-7e18a781609b/bond-cni-plugin/0.log"
Jul 29 15:32:42.771161 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:42.771108    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nwj5f_e666f634-f60c-47fb-952c-7e18a781609b/routeoverride-cni/0.log"
Jul 29 15:32:42.791285 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:42.791258    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nwj5f_e666f634-f60c-47fb-952c-7e18a781609b/whereabouts-cni-bincopy/0.log"
Jul 29 15:32:42.811603 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:42.811575    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nwj5f_e666f634-f60c-47fb-952c-7e18a781609b/whereabouts-cni/0.log"
Jul 29 15:32:43.009761 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:43.009721    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jsfjb_ede7f878-5cc1-457e-8849-8ed44e5945bb/kube-multus/0.log"
Jul 29 15:32:43.148238 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:43.148206    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vslsw_ad392a5c-af96-47d5-acdd-9905fc1cf0d5/network-metrics-daemon/0.log"
Jul 29 15:32:43.168237 ip-10-0-136-196 kubenswrapper[2563]: I0729 15:32:43.168208    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vslsw_ad392a5c-af96-47d5-acdd-9905fc1cf0d5/kube-rbac-proxy/0.log"