Jul 28 22:05:02.507648 ip-10-0-128-37 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 22:05:02.884932 ip-10-0-128-37 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 22:05:02.884932 ip-10-0-128-37 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 22:05:02.884932 ip-10-0-128-37 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 22:05:02.884932 ip-10-0-128-37 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 22:05:02.884932 ip-10-0-128-37 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 22:05:02.887781 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.887692    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 22:05:02.891927 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.891911    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 22:05:02.891927 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.891925    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 22:05:02.891927 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.891929    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 22:05:02.892021 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.891932    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 22:05:02.892021 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.891935    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 22:05:02.892021 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.891940    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 22:05:02.892021 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.891944    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 22:05:02.892021 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.891947    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 22:05:02.892021 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.891950    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 22:05:02.892021 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.891953    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 22:05:02.892021 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.891956    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 22:05:02.892021 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.891959    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 22:05:02.892021 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.891962    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 22:05:02.892021 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.891964    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 22:05:02.892021 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.891968    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 22:05:02.892021 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.891971    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 22:05:02.892021 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.891974    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 22:05:02.892021 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.891976    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 22:05:02.892021 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.891979    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 22:05:02.892021 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.891982    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 22:05:02.892021 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.891984    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 22:05:02.892021 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.891987    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 22:05:02.892538 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.891989    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 22:05:02.892538 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.891991    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 22:05:02.892538 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.891994    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 22:05:02.892538 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.891997    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 22:05:02.892538 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.891999    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 22:05:02.892538 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892002    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 22:05:02.892538 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892005    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 22:05:02.892538 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892007    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 22:05:02.892538 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892010    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 22:05:02.892538 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892013    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 22:05:02.892538 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892015    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 22:05:02.892538 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892018    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 22:05:02.892538 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892020    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 22:05:02.892538 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892023    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 22:05:02.892538 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892026    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 22:05:02.892538 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892029    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 22:05:02.892538 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892032    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 22:05:02.892538 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892034    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 22:05:02.892538 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892037    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 22:05:02.892538 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892040    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 22:05:02.893016 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892043    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 22:05:02.893016 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892045    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 22:05:02.893016 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892048    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 22:05:02.893016 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892050    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 22:05:02.893016 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892053    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 22:05:02.893016 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892056    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 22:05:02.893016 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892058    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 22:05:02.893016 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892060    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 22:05:02.893016 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892063    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 22:05:02.893016 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892065    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 22:05:02.893016 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892068    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 22:05:02.893016 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892070    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 22:05:02.893016 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892073    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 22:05:02.893016 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892077    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 22:05:02.893016 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892079    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 22:05:02.893016 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892082    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 22:05:02.893016 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892085    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 22:05:02.893016 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892087    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 22:05:02.893016 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892090    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 22:05:02.893016 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892093    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 22:05:02.893527 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892095    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 22:05:02.893527 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892098    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 22:05:02.893527 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892101    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 22:05:02.893527 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892103    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 22:05:02.893527 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892106    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 22:05:02.893527 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892110    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 22:05:02.893527 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892112    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 22:05:02.893527 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892116    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 22:05:02.893527 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892118    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 22:05:02.893527 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892121    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 22:05:02.893527 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892124    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 22:05:02.893527 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892127    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 22:05:02.893527 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892131    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 22:05:02.893527 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892135    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 22:05:02.893527 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892138    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 22:05:02.893527 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892141    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 22:05:02.893527 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892144    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 22:05:02.893527 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892147    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 22:05:02.893527 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892149    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 22:05:02.893983 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892153    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 22:05:02.893983 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892158    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 22:05:02.893983 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892161    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 22:05:02.893983 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892164    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 22:05:02.893983 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892166    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 22:05:02.893983 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892169    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 22:05:02.893983 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892172    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 22:05:02.893983 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892176    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 22:05:02.893983 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892178    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 22:05:02.893983 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892181    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 22:05:02.893983 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892184    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 22:05:02.893983 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892186    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 22:05:02.893983 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892189    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 22:05:02.893983 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892192    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 22:05:02.893983 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892194    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 22:05:02.893983 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892197    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 22:05:02.893983 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892200    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 22:05:02.893983 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892202    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 22:05:02.893983 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892921    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 22:05:02.894526 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892928    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 22:05:02.894526 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892931    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 22:05:02.894526 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892934    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 22:05:02.894526 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892937    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 22:05:02.894526 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892940    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 22:05:02.894526 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892943    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 22:05:02.894526 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892946    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 22:05:02.894526 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892948    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 22:05:02.894526 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892951    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 22:05:02.894526 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892954    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 22:05:02.894526 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892956    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 22:05:02.894526 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892959    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 22:05:02.894526 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892961    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 22:05:02.894526 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892964    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 22:05:02.894526 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892967    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 22:05:02.894526 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892969    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 22:05:02.894526 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892972    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 22:05:02.894526 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892974    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 22:05:02.894526 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892977    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 22:05:02.895067 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892980    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 22:05:02.895067 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892983    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 22:05:02.895067 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892986    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 22:05:02.895067 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892988    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 22:05:02.895067 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892991    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 22:05:02.895067 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892994    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 22:05:02.895067 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892996    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 22:05:02.895067 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.892999    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 22:05:02.895067 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893001    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 22:05:02.895067 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893004    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 22:05:02.895067 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893006    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 22:05:02.895067 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893008    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 22:05:02.895067 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893011    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 22:05:02.895067 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893014    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 22:05:02.895067 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893016    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 22:05:02.895067 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893019    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 22:05:02.895067 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893022    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 22:05:02.895067 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893025    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 22:05:02.895067 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893028    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 22:05:02.895067 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893031    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 22:05:02.895580 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893033    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 22:05:02.895580 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893036    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 22:05:02.895580 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893039    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 22:05:02.895580 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893041    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 22:05:02.895580 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893044    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 22:05:02.895580 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893046    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 22:05:02.895580 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893049    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 22:05:02.895580 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893054    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 22:05:02.895580 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893058    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 22:05:02.895580 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893061    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 22:05:02.895580 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893064    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 22:05:02.895580 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893067    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 22:05:02.895580 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893070    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 22:05:02.895580 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893073    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 22:05:02.895580 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893075    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 22:05:02.895580 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893078    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 22:05:02.895580 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893082    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 22:05:02.895580 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893085    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 22:05:02.895580 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893088    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 22:05:02.895580 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893090    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 22:05:02.896059 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893093    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 22:05:02.896059 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893096    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 22:05:02.896059 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893098    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 22:05:02.896059 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893101    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 22:05:02.896059 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893103    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 22:05:02.896059 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893106    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 22:05:02.896059 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893109    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 22:05:02.896059 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893112    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 22:05:02.896059 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893114    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 22:05:02.896059 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893117    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 22:05:02.896059 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893121    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 22:05:02.896059 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893125    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 22:05:02.896059 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893128    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 22:05:02.896059 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893131    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 22:05:02.896059 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893134    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 22:05:02.896059 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893136    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 22:05:02.896059 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893139    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 22:05:02.896059 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893142    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 22:05:02.896059 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893144    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 22:05:02.896533 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893147    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 22:05:02.896533 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893150    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 22:05:02.896533 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893152    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 22:05:02.896533 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893154    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 22:05:02.896533 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893158    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 22:05:02.896533 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893161    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 22:05:02.896533 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893163    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 22:05:02.896533 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893166    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 22:05:02.896533 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893168    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 22:05:02.896533 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893171    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 22:05:02.896533 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893174    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 22:05:02.896533 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893176    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 22:05:02.896533 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893180    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 22:05:02.896533 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893183    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 22:05:02.896533 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893186    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 22:05:02.896533 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893189    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 22:05:02.896533 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893191    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 22:05:02.896533 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893194    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 22:05:02.896533 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893197    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 22:05:02.896979 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.893200    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 22:05:02.896979 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894264    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 22:05:02.896979 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894271    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 22:05:02.896979 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894276    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 22:05:02.896979 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894279    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 22:05:02.896979 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894282    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 22:05:02.896979 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894285    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 22:05:02.896979 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894288    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 22:05:02.896979 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894291    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 22:05:02.896979 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894294    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 22:05:02.896979 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894297    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 22:05:02.896979 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894300    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 22:05:02.896979 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894302    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 22:05:02.896979 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894305    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 22:05:02.896979 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894308    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 22:05:02.896979 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894310    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 22:05:02.896979 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894313    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 22:05:02.896979 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894316    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 22:05:02.896979 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894318    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 22:05:02.896979 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894321    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 22:05:02.897500 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894324    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 22:05:02.897500 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894326    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 22:05:02.897500 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894329    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 22:05:02.897500 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894332    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 22:05:02.897500 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894334    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 22:05:02.897500 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894337    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 22:05:02.897500 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894340    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 22:05:02.897500 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894343    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 22:05:02.897500 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894345    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 22:05:02.897500 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894366    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 22:05:02.897500 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894370    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 22:05:02.897500 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894374    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 22:05:02.897500 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894378    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 22:05:02.897500 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894381    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 22:05:02.897500 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894384    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 22:05:02.897500 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894386    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 22:05:02.897500 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894389    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 22:05:02.897500 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894391    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 22:05:02.897500 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894400    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 22:05:02.897958 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894403    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 22:05:02.897958 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894406    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 22:05:02.897958 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894408    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 22:05:02.897958 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894411    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 22:05:02.897958 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894414    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 22:05:02.897958 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894418    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 22:05:02.897958 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894421    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 22:05:02.897958 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894424    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 22:05:02.897958 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894427    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 22:05:02.897958 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894430    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 22:05:02.897958 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894432    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 22:05:02.897958 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894435    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 22:05:02.897958 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894438    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 22:05:02.897958 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894440    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 22:05:02.897958 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894443    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 22:05:02.897958 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894447    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 22:05:02.897958 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894450    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 22:05:02.897958 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894452    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 22:05:02.897958 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894455    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 22:05:02.898432 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894457    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 22:05:02.898432 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894460    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 22:05:02.898432 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894462    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 22:05:02.898432 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894465    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 22:05:02.898432 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894467    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 22:05:02.898432 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894470    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 22:05:02.898432 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894472    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 22:05:02.898432 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894475    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 22:05:02.898432 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894477    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 22:05:02.898432 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894480    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 22:05:02.898432 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894482    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 22:05:02.898432 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894485    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 22:05:02.898432 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894488    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 22:05:02.898432 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894491    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 22:05:02.898432 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894493    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 22:05:02.898432 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894495    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 22:05:02.898432 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894498    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 22:05:02.898432 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894501    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 22:05:02.898432 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894503    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 22:05:02.898432 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894506    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 22:05:02.898951 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894513    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 22:05:02.898951 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894516    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 22:05:02.898951 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894519    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 22:05:02.898951 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894521    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 22:05:02.898951 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894524    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 22:05:02.898951 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894526    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 22:05:02.898951 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894529    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 22:05:02.898951 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894532    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 22:05:02.898951 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894535    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 22:05:02.898951 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894538    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 22:05:02.898951 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894541    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 22:05:02.898951 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894543    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 22:05:02.898951 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894546    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 22:05:02.898951 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894549    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 22:05:02.898951 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894552    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 22:05:02.898951 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894554    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 22:05:02.898951 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894557    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 22:05:02.898951 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894560    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 22:05:02.898951 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894562    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 22:05:02.898951 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894565    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 22:05:02.899474 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894568    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 22:05:02.899474 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.894571    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 22:05:02.899474 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894634    2563 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 22:05:02.899474 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894646    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 22:05:02.899474 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894652    2563 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 22:05:02.899474 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894656    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 22:05:02.899474 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894661    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 22:05:02.899474 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894666    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 22:05:02.899474 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894675    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 22:05:02.899474 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894682    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 22:05:02.899474 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894687    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 22:05:02.899474 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894694    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 22:05:02.899474 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894698    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 22:05:02.899474 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894703    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 22:05:02.899474 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894706    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 22:05:02.899474 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894709    2563 flags.go:64] FLAG: --cgroup-root=""
Jul 28 22:05:02.899474 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894712    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 22:05:02.899474 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894715    2563 flags.go:64] FLAG: --client-ca-file=""
Jul 28 22:05:02.899474 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894719    2563 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 22:05:02.899474 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894722    2563 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 22:05:02.899474 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894727    2563 flags.go:64] FLAG: --cluster-domain=""
Jul 28 22:05:02.899474 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894730    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 22:05:02.899474 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894733    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 22:05:02.899474 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894736    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 22:05:02.900045 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894740    2563 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 22:05:02.900045 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894744    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 22:05:02.900045 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894747    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 22:05:02.900045 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894750    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 22:05:02.900045 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894753    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 22:05:02.900045 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894758    2563 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 22:05:02.900045 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894761    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 22:05:02.900045 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894764    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 22:05:02.900045 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894767    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 22:05:02.900045 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894770    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 22:05:02.900045 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894774    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 22:05:02.900045 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894777    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 22:05:02.900045 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894780    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 22:05:02.900045 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894783    2563 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 22:05:02.900045 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894786    2563 flags.go:64] FLAG: --enable-server="true"
Jul 28 22:05:02.900045 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894789    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 22:05:02.900045 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894795    2563 flags.go:64] FLAG: --event-burst="100"
Jul 28 22:05:02.900045 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894798    2563 flags.go:64] FLAG: --event-qps="50"
Jul 28 22:05:02.900045 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894801    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 22:05:02.900045 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894804    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 22:05:02.900045 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894807    2563 flags.go:64] FLAG: --eviction-hard=""
Jul 28 22:05:02.900045 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894811    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 22:05:02.900045 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894814    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 22:05:02.900045 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894817    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 22:05:02.900045 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894820    2563 flags.go:64] FLAG: --eviction-soft=""
Jul 28 22:05:02.900685 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894823    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 22:05:02.900685 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894826    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 22:05:02.900685 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894829    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 22:05:02.900685 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894831    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 22:05:02.900685 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894834    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 22:05:02.900685 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894837    2563 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 22:05:02.900685 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894840    2563 flags.go:64] FLAG: --feature-gates=""
Jul 28 22:05:02.900685 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894844    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 22:05:02.900685 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894847    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 22:05:02.900685 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894850    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 22:05:02.900685 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894853    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 22:05:02.900685 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894856    2563 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 22:05:02.900685 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894860    2563 flags.go:64] FLAG: --help="false"
Jul 28 22:05:02.900685 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894863    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-128-37.ec2.internal"
Jul 28 22:05:02.900685 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894866    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 22:05:02.900685 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894868    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 22:05:02.900685 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894872    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 22:05:02.900685 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894875    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 22:05:02.900685 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894878    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 22:05:02.900685 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894881    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 22:05:02.900685 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894884    2563 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 22:05:02.900685 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894887    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 22:05:02.900685 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894890    2563 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 22:05:02.901274 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894895    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 22:05:02.901274 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894898    2563 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 22:05:02.901274 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894901    2563 flags.go:64] FLAG: --kube-reserved=""
Jul 28 22:05:02.901274 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894904    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 22:05:02.901274 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894906    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 22:05:02.901274 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894909    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 22:05:02.901274 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894912    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 22:05:02.901274 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894915    2563 flags.go:64] FLAG: --lock-file=""
Jul 28 22:05:02.901274 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894917    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 22:05:02.901274 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894920    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 22:05:02.901274 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894923    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 22:05:02.901274 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894928    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 22:05:02.901274 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894931    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 22:05:02.901274 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894935    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 22:05:02.901274 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894937    2563 flags.go:64] FLAG: --logging-format="text"
Jul 28 22:05:02.901274 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894941    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 22:05:02.901274 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894944    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 22:05:02.901274 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894947    2563 flags.go:64] FLAG: --manifest-url=""
Jul 28 22:05:02.901274 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894950    2563 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 22:05:02.901274 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894954    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 22:05:02.901274 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894957    2563 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 22:05:02.901274 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894962    2563 flags.go:64] FLAG: --max-pods="110"
Jul 28 22:05:02.901274 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894965    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 22:05:02.901274 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894968    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 22:05:02.901274 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894970    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 22:05:02.901899 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894974    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 22:05:02.901899 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894976    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 22:05:02.901899 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894979    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 22:05:02.901899 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894982    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 22:05:02.901899 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894989    2563 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 22:05:02.901899 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894992    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 22:05:02.901899 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.894995    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 22:05:02.901899 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895000    2563 flags.go:64] FLAG: --pod-cidr=""
Jul 28 22:05:02.901899 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895003    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 22:05:02.901899 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895008    2563 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 22:05:02.901899 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895012    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 22:05:02.901899 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895015    2563 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 22:05:02.901899 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895018    2563 flags.go:64] FLAG: --port="10250"
Jul 28 22:05:02.901899 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895021    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 22:05:02.901899 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895024    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0100bfa729cd21b7f"
Jul 28 22:05:02.901899 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895027    2563 flags.go:64] FLAG: --qos-reserved=""
Jul 28 22:05:02.901899 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895030    2563 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 22:05:02.901899 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895033    2563 flags.go:64] FLAG: --register-node="true"
Jul 28 22:05:02.901899 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895036    2563 flags.go:64] FLAG: --register-with-taints=""
Jul 28 22:05:02.901899 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895039    2563 flags.go:64] FLAG: --registry-burst="10"
Jul 28 22:05:02.901899 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895042    2563 flags.go:64] FLAG: --registry-qps="5"
Jul 28 22:05:02.901899 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895045    2563 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 22:05:02.901899 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895048    2563 flags.go:64] FLAG: --reserved-memory=""
Jul 28 22:05:02.901899 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895051    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 22:05:02.901899 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895054    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 22:05:02.902520 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895057    2563 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 22:05:02.902520 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895060    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 22:05:02.902520 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895063    2563 flags.go:64] FLAG: --runonce="false"
Jul 28 22:05:02.902520 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895067    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 22:05:02.902520 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895070    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 22:05:02.902520 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895073    2563 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 22:05:02.902520 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895076    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 22:05:02.902520 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895079    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 22:05:02.902520 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895082    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 22:05:02.902520 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895086    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 22:05:02.902520 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895089    2563 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 22:05:02.902520 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895092    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 22:05:02.902520 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895094    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 22:05:02.902520 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895097    2563 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 22:05:02.902520 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895100    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 22:05:02.902520 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895104    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 22:05:02.902520 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895108    2563 flags.go:64] FLAG: --system-cgroups=""
Jul 28 22:05:02.902520 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895110    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 22:05:02.902520 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895116    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 22:05:02.902520 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895119    2563 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 22:05:02.902520 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895122    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 22:05:02.902520 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895126    2563 flags.go:64] FLAG: --tls-min-version=""
Jul 28 22:05:02.902520 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895129    2563 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 22:05:02.902520 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895132    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 22:05:02.902520 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895134    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 22:05:02.903108 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895137    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 22:05:02.903108 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895140    2563 flags.go:64] FLAG: --v="2"
Jul 28 22:05:02.903108 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895144    2563 flags.go:64] FLAG: --version="false"
Jul 28 22:05:02.903108 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895150    2563 flags.go:64] FLAG: --vmodule=""
Jul 28 22:05:02.903108 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895154    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 22:05:02.903108 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895157    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 22:05:02.903108 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895275    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 22:05:02.903108 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895279    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 22:05:02.903108 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895282    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 22:05:02.903108 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895285    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 22:05:02.903108 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895289    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 22:05:02.903108 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895292    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 22:05:02.903108 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895295    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 22:05:02.903108 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895298    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 22:05:02.903108 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895301    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 22:05:02.903108 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895304    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 22:05:02.903108 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895307    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 22:05:02.903108 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895310    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 22:05:02.903108 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895312    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 22:05:02.903108 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895315    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 22:05:02.903108 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895318    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 22:05:02.903692 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895321    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 22:05:02.903692 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895324    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 22:05:02.903692 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895327    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 22:05:02.903692 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895330    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 22:05:02.903692 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895333    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 22:05:02.903692 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895335    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 22:05:02.903692 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895338    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 22:05:02.903692 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895341    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 22:05:02.903692 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895343    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 22:05:02.903692 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895346    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 22:05:02.903692 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895361    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 22:05:02.903692 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895364    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 22:05:02.903692 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895366    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 22:05:02.903692 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895369    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 22:05:02.903692 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895372    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 22:05:02.903692 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895374    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 22:05:02.903692 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895377    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 22:05:02.903692 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895380    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 22:05:02.903692 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895383    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 22:05:02.904157 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895385    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 22:05:02.904157 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895388    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 22:05:02.904157 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895391    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 22:05:02.904157 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895394    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 22:05:02.904157 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895397    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 22:05:02.904157 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895399    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 22:05:02.904157 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895402    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 22:05:02.904157 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895405    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 22:05:02.904157 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895415    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 22:05:02.904157 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895419    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 22:05:02.904157 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895422    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 22:05:02.904157 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895426    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 22:05:02.904157 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895430    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 22:05:02.904157 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895433    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 22:05:02.904157 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895438    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 22:05:02.904157 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895441    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 22:05:02.904157 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895444    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 22:05:02.904157 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895449    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 22:05:02.904157 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895452    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 22:05:02.904694 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895455    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 22:05:02.904694 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895458    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 22:05:02.904694 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895461    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 22:05:02.904694 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895463    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 22:05:02.904694 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895466    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 22:05:02.904694 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895469    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 22:05:02.904694 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895472    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 22:05:02.904694 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895474    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 22:05:02.904694 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895477    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 22:05:02.904694 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895480    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 22:05:02.904694 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895482    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 22:05:02.904694 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895485    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 22:05:02.904694 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895488    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 22:05:02.904694 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895490    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 22:05:02.904694 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895493    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 22:05:02.904694 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895497    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 22:05:02.904694 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895500    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 22:05:02.904694 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895503    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 22:05:02.904694 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895506    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 22:05:02.904694 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895508    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 22:05:02.905178 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895512    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 22:05:02.905178 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895514    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 22:05:02.905178 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895517    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 22:05:02.905178 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895519    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 22:05:02.905178 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895522    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 22:05:02.905178 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895525    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 22:05:02.905178 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895527    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 22:05:02.905178 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895546    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 22:05:02.905178 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895549    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 22:05:02.905178 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895553    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 22:05:02.905178 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895556    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 22:05:02.905178 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895559    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 22:05:02.905178 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895563    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 22:05:02.905178 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895565    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 22:05:02.905178 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895568    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 22:05:02.905178 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895571    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 22:05:02.905178 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895574    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 22:05:02.905178 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895576    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 22:05:02.905178 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895579    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 22:05:02.905178 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895581    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 22:05:02.905686 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895584    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 22:05:02.905686 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895587    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 22:05:02.905686 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895590    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 22:05:02.905686 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895592    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 22:05:02.905686 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895595    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 22:05:02.905686 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.895598    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 22:05:02.905686 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.895606    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 22:05:02.905686 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.902576    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 22:05:02.905686 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.902591    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 22:05:02.905686 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.902609    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 22:05:02.905686 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.902615    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 22:05:02.905686 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902677    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 22:05:02.905686 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902682    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 22:05:02.905686 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902685    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 22:05:02.906042 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902689    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 22:05:02.906042 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902692    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 22:05:02.906042 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902696    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 22:05:02.906042 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902701    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 22:05:02.906042 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902704    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 22:05:02.906042 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902707    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 22:05:02.906042 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902710    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 22:05:02.906042 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902712    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 22:05:02.906042 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902715    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 22:05:02.906042 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902718    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 22:05:02.906042 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902721    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 22:05:02.906042 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902724    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 22:05:02.906042 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902728    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 22:05:02.906042 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902731    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 22:05:02.906042 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902734    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 22:05:02.906042 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902737    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 22:05:02.906042 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902740    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 22:05:02.906042 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902743    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 22:05:02.906042 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902746    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 22:05:02.906513 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902749    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 22:05:02.906513 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902752    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 22:05:02.906513 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902754    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 22:05:02.906513 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902757    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 22:05:02.906513 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902760    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 22:05:02.906513 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902763    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 22:05:02.906513 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902765    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 22:05:02.906513 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902768    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 22:05:02.906513 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902771    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 22:05:02.906513 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902774    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 22:05:02.906513 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902777    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 22:05:02.906513 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902779    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 22:05:02.906513 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902782    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 22:05:02.906513 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902784    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 22:05:02.906513 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902787    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 22:05:02.906513 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902790    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 22:05:02.906513 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902793    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 22:05:02.906513 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902795    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 22:05:02.906513 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902798    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 22:05:02.906513 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902801    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 22:05:02.906985 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902804    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 22:05:02.906985 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902807    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 22:05:02.906985 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902809    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 22:05:02.906985 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902818    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 22:05:02.906985 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902821    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 22:05:02.906985 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902824    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 22:05:02.906985 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902826    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 22:05:02.906985 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902829    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 22:05:02.906985 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902831    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 22:05:02.906985 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902834    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 22:05:02.906985 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902836    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 22:05:02.906985 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902839    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 22:05:02.906985 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902842    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 22:05:02.906985 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902844    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 22:05:02.906985 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902847    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 22:05:02.906985 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902850    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 22:05:02.906985 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902852    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 22:05:02.906985 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902855    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 22:05:02.906985 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902857    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 22:05:02.906985 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902860    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 22:05:02.907487 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902863    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 22:05:02.907487 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902866    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 22:05:02.907487 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902868    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 22:05:02.907487 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902871    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 22:05:02.907487 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902874    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 22:05:02.907487 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902876    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 22:05:02.907487 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902879    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 22:05:02.907487 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902882    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 22:05:02.907487 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902885    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 22:05:02.907487 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902887    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 22:05:02.907487 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902890    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 22:05:02.907487 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902892    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 22:05:02.907487 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902895    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 22:05:02.907487 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902898    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 22:05:02.907487 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902900    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 22:05:02.907487 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902902    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 22:05:02.907487 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902910    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 22:05:02.907487 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902913    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 22:05:02.907487 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902915    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 22:05:02.907487 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902918    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 22:05:02.907971 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902920    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 22:05:02.907971 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902922    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 22:05:02.907971 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902925    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 22:05:02.907971 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902927    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 22:05:02.907971 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902930    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 22:05:02.907971 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902932    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 22:05:02.907971 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902934    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 22:05:02.907971 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902938    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 22:05:02.907971 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902940    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 22:05:02.907971 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902943    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 22:05:02.907971 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902947    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 22:05:02.907971 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902951    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 22:05:02.907971 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902954    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 22:05:02.907971 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902957    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 22:05:02.907971 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902960    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 22:05:02.907971 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902963    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 22:05:02.907971 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.902966    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 22:05:02.908478 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.902971    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 22:05:02.908478 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903119    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 22:05:02.908478 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903124    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 22:05:02.908478 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903126    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 22:05:02.908478 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903129    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 22:05:02.908478 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903132    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 22:05:02.908478 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903135    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 22:05:02.908478 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903138    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 22:05:02.908478 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903142    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 22:05:02.908478 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903145    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 22:05:02.908478 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903147    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 22:05:02.908478 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903150    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 22:05:02.908478 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903158    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 22:05:02.908478 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903161    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 22:05:02.908821 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903163    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 22:05:02.908821 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903166    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 22:05:02.908821 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903170    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 22:05:02.908821 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903173    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 22:05:02.908821 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903176    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 22:05:02.908821 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903179    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 22:05:02.908821 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903181    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 22:05:02.908821 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903184    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 22:05:02.908821 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903186    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 22:05:02.908821 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903190    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 22:05:02.908821 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903193    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 22:05:02.908821 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903195    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 22:05:02.908821 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903198    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 22:05:02.908821 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903200    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 22:05:02.908821 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903203    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 22:05:02.908821 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903205    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 22:05:02.908821 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903208    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 22:05:02.908821 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903210    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 22:05:02.908821 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903213    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 22:05:02.909270 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903215    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 22:05:02.909270 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903218    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 22:05:02.909270 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903220    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 22:05:02.909270 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903223    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 22:05:02.909270 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903225    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 22:05:02.909270 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903228    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 22:05:02.909270 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903230    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 22:05:02.909270 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903233    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 22:05:02.909270 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903235    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 22:05:02.909270 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903238    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 22:05:02.909270 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903240    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 22:05:02.909270 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903242    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 22:05:02.909270 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903251    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 22:05:02.909270 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903255    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 22:05:02.909270 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903258    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 22:05:02.909270 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903261    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 22:05:02.909270 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903264    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 22:05:02.909270 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903266    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 22:05:02.909270 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903269    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 22:05:02.909741 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903272    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 22:05:02.909741 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903275    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 22:05:02.909741 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903277    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 22:05:02.909741 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903280    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 22:05:02.909741 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903283    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 22:05:02.909741 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903286    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 22:05:02.909741 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903289    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 22:05:02.909741 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903291    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 22:05:02.909741 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903294    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 22:05:02.909741 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903296    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 22:05:02.909741 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903299    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 22:05:02.909741 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903301    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 22:05:02.909741 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903304    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 22:05:02.909741 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903306    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 22:05:02.909741 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903309    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 22:05:02.909741 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903312    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 22:05:02.909741 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903320    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 22:05:02.909741 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903322    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 22:05:02.909741 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903325    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 22:05:02.909741 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903327    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 22:05:02.910217 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903330    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 22:05:02.910217 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903332    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 22:05:02.910217 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903335    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 22:05:02.910217 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903337    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 22:05:02.910217 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903340    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 22:05:02.910217 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903342    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 22:05:02.910217 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903368    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 22:05:02.910217 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903372    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 22:05:02.910217 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903375    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 22:05:02.910217 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903377    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 22:05:02.910217 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903380    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 22:05:02.910217 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903382    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 22:05:02.910217 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903385    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 22:05:02.910217 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903388    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 22:05:02.910217 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903390    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 22:05:02.910217 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903393    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 22:05:02.910217 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903395    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 22:05:02.910217 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903398    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 22:05:02.910217 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903401    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 22:05:02.910217 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903403    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 22:05:02.910705 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903406    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 22:05:02.910705 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903418    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 22:05:02.910705 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903421    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 22:05:02.910705 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903424    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 22:05:02.910705 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903426    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 22:05:02.910705 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903429    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 22:05:02.910705 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903432    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 22:05:02.910705 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:02.903434    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 22:05:02.910705 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.903439    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 22:05:02.910705 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.903553    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 22:05:02.910705 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.905703    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 22:05:02.910705 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.906987    2563 server.go:1019] "Starting client certificate rotation"
Jul 28 22:05:02.910705 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.907096    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 22:05:02.910705 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.907896    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 22:05:02.911046 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.910128    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 22:05:02.916768 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.916745    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 22:05:02.926311 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.926292    2563 log.go:25] "Validated CRI v1 runtime API"
Jul 28 22:05:02.928466 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.928450    2563 log.go:25] "Validated CRI v1 image API"
Jul 28 22:05:02.929546 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.929514    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 22:05:02.932010 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.931986    2563 fs.go:135] Filesystem UUIDs: map[3a9405f3-ea72-4a7d-a298-c39d6ca42560:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 8e08025f-634f-4c84-ae5e-97ad070dfaf5:/dev/nvme0n1p3]
Jul 28 22:05:02.932091 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.932010    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 22:05:02.936802 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.936686    2563 manager.go:216] Machine: {Timestamp:2026-07-28 22:05:02.936148241 +0000 UTC m=+0.316285876 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099710 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24a4a6986a101e595fc79ebd5c9345 SystemUUID:ec24a4a6-986a-101e-595f-c79ebd5c9345 BootID:07e5576c-b069-49a9-aa8a-728a0a30a05d Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:57:b1:ff:3c:a9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:57:b1:ff:3c:a9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:09:36:8d:44:bf Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 22:05:02.936802 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.936793    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 22:05:02.936933 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.936900    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 22:05:02.938934 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.938915    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 22:05:02.939852 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.939828    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 22:05:02.939984 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.939854    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-37.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 22:05:02.940625 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.940616    2563 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 22:05:02.940663 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.940627    2563 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 22:05:02.940663 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.940634    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 22:05:02.941275 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.941264    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 22:05:02.942003 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.941994    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 22:05:02.943622 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.943610    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 22:05:02.943755 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.943745    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 22:05:02.943787 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.943774    2563 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 22:05:02.943787 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.943784    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 22:05:02.943840 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.943800    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 22:05:02.943840 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.943809    2563 kubelet.go:398] "Adding apiserver pod source"
Jul 28 22:05:02.943840 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.943819    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 22:05:02.944157 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.944145    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 22:05:02.944208 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.944163    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 22:05:02.946113 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.946092    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 22:05:02.947830 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.947814    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 22:05:02.947910 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.947844    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 22:05:02.949523 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.949512    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 22:05:02.949574 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.949527    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 22:05:02.949574 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.949534    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 22:05:02.949574 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.949540    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 22:05:02.949574 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.949548    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 22:05:02.949574 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.949553    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 22:05:02.949574 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.949561    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 22:05:02.949574 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.949567    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 22:05:02.949574 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.949573    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 22:05:02.949770 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.949579    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 22:05:02.949770 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.949588    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 22:05:02.949770 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.949593    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 22:05:02.950934 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.950923    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 22:05:02.950934 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.950933    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 22:05:02.953667 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:02.953644    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-37.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 22:05:02.953667 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:02.953645    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 22:05:02.953815 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.953686    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-37.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 22:05:02.954432 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.954413    2563 server.go:1268] "Started kubelet"
Jul 28 22:05:02.955212 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.955118    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 22:05:02.955212 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.955131    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 22:05:02.955272 ip-10-0-128-37 systemd[1]: Started Kubernetes Kubelet.
Jul 28 22:05:02.955634 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.955612    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 22:05:02.957150 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.957130    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 22:05:02.957637 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.957606    2563 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 22:05:02.960061 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.960039    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9s57x"
Jul 28 22:05:02.960695 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:02.960676    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 22:05:02.962442 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:02.961586    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-37.ec2.internal.18c69356b784a6c8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-37.ec2.internal,UID:ip-10-0-128-37.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-37.ec2.internal,},FirstTimestamp:2026-07-28 22:05:02.954374856 +0000 UTC m=+0.334512491,LastTimestamp:2026-07-28 22:05:02.954374856 +0000 UTC m=+0.334512491,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-37.ec2.internal,}"
Jul 28 22:05:02.965759 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.965742    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9s57x"
Jul 28 22:05:02.965826 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.965767    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 22:05:02.965878 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.965834    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 22:05:02.965934 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:02.965905    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-37.ec2.internal\" not found"
Jul 28 22:05:02.966558 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.966540    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 22:05:02.966558 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.966540    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 22:05:02.966708 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.966567    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 22:05:02.966708 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.966608    2563 factory.go:55] Registering systemd factory
Jul 28 22:05:02.966708 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.966622    2563 factory.go:223] Registration of the systemd container factory successfully
Jul 28 22:05:02.966708 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.966638    2563 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 22:05:02.966708 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.966647    2563 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 22:05:02.966897 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.966851    2563 factory.go:153] Registering CRI-O factory
Jul 28 22:05:02.966897 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.966865    2563 factory.go:223] Registration of the crio container factory successfully
Jul 28 22:05:02.966991 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.966916    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 22:05:02.966991 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.966937    2563 factory.go:103] Registering Raw factory
Jul 28 22:05:02.966991 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.966952    2563 manager.go:1196] Started watching for new ooms in manager
Jul 28 22:05:02.967402 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.967390    2563 manager.go:349] Starting recovery of all containers
Jul 28 22:05:02.974316 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.974295    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 22:05:02.976215 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:02.976192    2563 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-128-37.ec2.internal\" not found" node="ip-10-0-128-37.ec2.internal"
Jul 28 22:05:02.978499 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.978483    2563 manager.go:354] Recovery completed
Jul 28 22:05:02.982891 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.982879    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 22:05:02.984775 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.984761    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 22:05:02.984829 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.984794    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 22:05:02.984829 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.984808    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasSufficientPID"
Jul 28 22:05:02.985726 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.985688    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 22:05:02.985726 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.985724    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 22:05:02.985861 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.985750    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 22:05:02.986871 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.986850    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 22:05:02.987459 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.987446    2563 policy_none.go:49] "None policy: Start"
Jul 28 22:05:02.987536 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.987467    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 22:05:02.987536 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.987482    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 22:05:02.987917 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.987898    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 22:05:02.988010 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.987936    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 22:05:02.988010 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.987951    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasSufficientPID"
Jul 28 22:05:02.988780 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:02.988767    2563 policy_none.go:47] "Start"
Jul 28 22:05:03.021014 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.020889    2563 manager.go:337] "Starting Device Plugin manager"
Jul 28 22:05:03.021097 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:03.021038    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 22:05:03.021097 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.021053    2563 server.go:87] "Starting device plugin registration server"
Jul 28 22:05:03.021307 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.021294    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 22:05:03.021387 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.021308    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 22:05:03.021444 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.021400    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 22:05:03.021494 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.021487    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 22:05:03.021537 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.021496    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 22:05:03.022025 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:03.022006    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 22:05:03.022093 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:03.022061    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-37.ec2.internal\" not found"
Jul 28 22:05:03.121874 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.121848    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 22:05:03.123071 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.122249    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 22:05:03.123071 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.122882    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 22:05:03.123071 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.122910    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 22:05:03.123071 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.122923    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasSufficientPID"
Jul 28 22:05:03.123071 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.122957    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-37.ec2.internal"
Jul 28 22:05:03.123648 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.123630    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 22:05:03.123740 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.123670    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 22:05:03.123740 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.123698    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 22:05:03.123740 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:03.123730    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 22:05:03.125335 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.125318    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 22:05:03.130872 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.130857    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-37.ec2.internal"
Jul 28 22:05:03.130947 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:03.130877    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-37.ec2.internal\": node \"ip-10-0-128-37.ec2.internal\" not found"
Jul 28 22:05:03.209130 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:03.209042    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-37.ec2.internal\" not found"
Jul 28 22:05:03.224143 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.224123    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-37.ec2.internal"]
Jul 28 22:05:03.224199 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.224183    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 22:05:03.225027 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.225010    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 22:05:03.225111 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.225039    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 22:05:03.225111 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.225048    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasSufficientPID"
Jul 28 22:05:03.226428 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.226416    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 22:05:03.226565 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.226537    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal"
Jul 28 22:05:03.226608 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.226571    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 22:05:03.227088 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.227065    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 22:05:03.227146 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.227094    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 22:05:03.227146 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.227106    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasSufficientPID"
Jul 28 22:05:03.227272 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.227065    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 22:05:03.227272 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.227174    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 22:05:03.227272 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.227186    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasSufficientPID"
Jul 28 22:05:03.228322 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.228305    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-37.ec2.internal"
Jul 28 22:05:03.228407 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.228336    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 22:05:03.228951 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.228938    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 22:05:03.229001 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.228960    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 22:05:03.229001 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.228969    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeHasSufficientPID"
Jul 28 22:05:03.243871 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:03.243853    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-37.ec2.internal\" not found" node="ip-10-0-128-37.ec2.internal"
Jul 28 22:05:03.247155 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:03.247140    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-37.ec2.internal\" not found" node="ip-10-0-128-37.ec2.internal"
Jul 28 22:05:03.268224 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.268205    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e5ce7b7b59f5b560ebbfcc3f6f472e5b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal\" (UID: \"e5ce7b7b59f5b560ebbfcc3f6f472e5b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal"
Jul 28 22:05:03.268307 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.268233    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5ce7b7b59f5b560ebbfcc3f6f472e5b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal\" (UID: \"e5ce7b7b59f5b560ebbfcc3f6f472e5b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal"
Jul 28 22:05:03.268307 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.268261    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2aaf76ae458a97a3e5edaa3e36196e52-config\") pod \"kube-apiserver-proxy-ip-10-0-128-37.ec2.internal\" (UID: \"2aaf76ae458a97a3e5edaa3e36196e52\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-37.ec2.internal"
Jul 28 22:05:03.309906 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:03.309885    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-37.ec2.internal\" not found"
Jul 28 22:05:03.369083 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.369060    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e5ce7b7b59f5b560ebbfcc3f6f472e5b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal\" (UID: \"e5ce7b7b59f5b560ebbfcc3f6f472e5b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal"
Jul 28 22:05:03.369201 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.369089    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5ce7b7b59f5b560ebbfcc3f6f472e5b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal\" (UID: \"e5ce7b7b59f5b560ebbfcc3f6f472e5b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal"
Jul 28 22:05:03.369201 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.369107    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2aaf76ae458a97a3e5edaa3e36196e52-config\") pod \"kube-apiserver-proxy-ip-10-0-128-37.ec2.internal\" (UID: \"2aaf76ae458a97a3e5edaa3e36196e52\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-37.ec2.internal"
Jul 28 22:05:03.369201 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.369160    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2aaf76ae458a97a3e5edaa3e36196e52-config\") pod \"kube-apiserver-proxy-ip-10-0-128-37.ec2.internal\" (UID: \"2aaf76ae458a97a3e5edaa3e36196e52\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-37.ec2.internal"
Jul 28 22:05:03.369201 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.369173    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e5ce7b7b59f5b560ebbfcc3f6f472e5b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal\" (UID: \"e5ce7b7b59f5b560ebbfcc3f6f472e5b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal"
Jul 28 22:05:03.369337 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.369205    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5ce7b7b59f5b560ebbfcc3f6f472e5b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal\" (UID: \"e5ce7b7b59f5b560ebbfcc3f6f472e5b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal"
Jul 28 22:05:03.410370 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:03.410310    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-37.ec2.internal\" not found"
Jul 28 22:05:03.513767 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:03.513690    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-37.ec2.internal\" not found"
Jul 28 22:05:03.548513 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.548469    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal"
Jul 28 22:05:03.550194 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.550176    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-37.ec2.internal"
Jul 28 22:05:03.614598 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:03.614560    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-37.ec2.internal\" not found"
Jul 28 22:05:03.717435 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:03.717402    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-37.ec2.internal\" not found"
Jul 28 22:05:03.820638 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:03.820562    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-37.ec2.internal\" not found"
Jul 28 22:05:03.865265 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.865244    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 22:05:03.908566 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.908537    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 22:05:03.908983 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.908669    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 22:05:03.908983 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.908713    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 22:05:03.908983 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.908718    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 22:05:03.920677 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:03.920651    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-37.ec2.internal\" not found"
Jul 28 22:05:03.970756 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.970723    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 22:00:03 +0000 UTC" deadline="2028-03-02 20:25:56.163064772 +0000 UTC"
Jul 28 22:05:03.970756 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:03.970750    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13990h20m52.192317681s"
Jul 28 22:05:04.023306 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:04.023280    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-37.ec2.internal\" not found"
Jul 28 22:05:04.127287 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:04.127203    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-37.ec2.internal\" not found"
Jul 28 22:05:04.231621 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:04.231574    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-37.ec2.internal\" not found"
Jul 28 22:05:04.336017 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:04.335978    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-37.ec2.internal\" not found"
Jul 28 22:05:04.437275 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:04.437197    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-37.ec2.internal\" not found"
Jul 28 22:05:04.494296 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.494269    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 22:05:04.566515 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.566487    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal"
Jul 28 22:05:04.581271 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.581253    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 22:05:04.582274 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.582256    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-37.ec2.internal"
Jul 28 22:05:04.592543 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.592520    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 22:05:04.666127 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:04.666094    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aaf76ae458a97a3e5edaa3e36196e52.slice/crio-727d8cc58ec803ee8420b78918afdb2570a7f44bcbacba55a631d44743c8b0e8 WatchSource:0}: Error finding container 727d8cc58ec803ee8420b78918afdb2570a7f44bcbacba55a631d44743c8b0e8: Status 404 returned error can't find the container with id 727d8cc58ec803ee8420b78918afdb2570a7f44bcbacba55a631d44743c8b0e8
Jul 28 22:05:04.672428 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.672405    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 22:05:04.878086 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:04.878055    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ce7b7b59f5b560ebbfcc3f6f472e5b.slice/crio-40e3ac3bc03e0ef16f82ff4bfb020216d2652df4a1bcebdfbd241d90d0ea9f92 WatchSource:0}: Error finding container 40e3ac3bc03e0ef16f82ff4bfb020216d2652df4a1bcebdfbd241d90d0ea9f92: Status 404 returned error can't find the container with id 40e3ac3bc03e0ef16f82ff4bfb020216d2652df4a1bcebdfbd241d90d0ea9f92
Jul 28 22:05:04.946145 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.946111    2563 apiserver.go:52] "Watching apiserver"
Jul 28 22:05:04.958141 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.958117    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 22:05:04.958514 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.958493    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-4xqhf","openshift-ovn-kubernetes/ovnkube-node-mcj8p","kube-system/global-pull-secret-syncer-2rqhg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqvb4","openshift-image-registry/node-ca-tx5m6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal","openshift-multus/multus-pkhlk","openshift-network-diagnostics/network-check-target-87zr2","openshift-network-operator/iptables-alerter-pwnwp","kube-system/konnectivity-agent-7wmm5","kube-system/kube-apiserver-proxy-ip-10-0-128-37.ec2.internal","openshift-cluster-node-tuning-operator/tuned-jhwgt","openshift-dns/node-resolver-qtgkr","openshift-multus/multus-additional-cni-plugins-4fjz6"]
Jul 28 22:05:04.960322 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.960306    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:04.961624 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.961601    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qtgkr"
Jul 28 22:05:04.963620 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.963600    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rqhg"
Jul 28 22:05:04.963778 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:04.963755    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rqhg" podUID="5d7dfdb2-a77a-47e8-865a-07a014ec5633"
Jul 28 22:05:04.964121 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.964103    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-lbzkj\""
Jul 28 22:05:04.964204 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.964136    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 22:05:04.964204 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.964185    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 22:05:04.964204 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.964198    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 22:05:04.964378 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.964243    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 22:05:04.964669 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.964653    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 22:05:04.964750 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.964681    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 22:05:04.965042 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.965029    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqvb4"
Jul 28 22:05:04.965778 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.965761    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cmsjt\""
Jul 28 22:05:04.966268 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.966252    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tx5m6"
Jul 28 22:05:04.966518 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.966503    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 22:05:04.967543 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.967525    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ldvzz\""
Jul 28 22:05:04.967981 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.967965    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4fjz6"
Jul 28 22:05:04.968653 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.968563    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 22:05:04.968653 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.968597    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 22:05:04.968949 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.968931    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 22:05:04.969362 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.969345    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pwnwp"
Jul 28 22:05:04.969823 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.969764    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-57bs2\""
Jul 28 22:05:04.971521 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.970599    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 22:05:04.971521 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.971093    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 22:05:04.971521 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.971215    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 22:05:04.971521 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.971361    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 22:05:04.972711 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.971963    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4xqhf"
Jul 28 22:05:04.972711 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:04.972164    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4xqhf" podUID="2f33e531-3341-4c01-940e-91f17519976c"
Jul 28 22:05:04.972711 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.972253    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dtfkc\""
Jul 28 22:05:04.972711 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.972311    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 22:05:04.972711 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.972592    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 22:05:04.974367 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.973676    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 22:05:04.974367 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.973836    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 22:05:04.974367 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.974065    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4bzzl\""
Jul 28 22:05:04.975397 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.975351    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:04.977053 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.977033    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7wmm5"
Jul 28 22:05:04.978028 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.978008    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 22:05:04.978159 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.978139    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52t5j\" (UniqueName: \"kubernetes.io/projected/820ee419-dab7-43ef-94bf-cadbaf9a5232-kube-api-access-52t5j\") pod \"node-resolver-qtgkr\" (UID: \"820ee419-dab7-43ef-94bf-cadbaf9a5232\") " pod="openshift-dns/node-resolver-qtgkr"
Jul 28 22:05:04.978245 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.978175    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5d7dfdb2-a77a-47e8-865a-07a014ec5633-kubelet-config\") pod \"global-pull-secret-syncer-2rqhg\" (UID: \"5d7dfdb2-a77a-47e8-865a-07a014ec5633\") " pod="kube-system/global-pull-secret-syncer-2rqhg"
Jul 28 22:05:04.978245 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.978203    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/caba445d-47e5-446e-9be4-9148e4d7e59a-registration-dir\") pod \"aws-ebs-csi-driver-node-hqvb4\" (UID: \"caba445d-47e5-446e-9be4-9148e4d7e59a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqvb4"
Jul 28 22:05:04.978245 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.978229    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 22:05:04.978245 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.978229    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/caba445d-47e5-446e-9be4-9148e4d7e59a-etc-selinux\") pod \"aws-ebs-csi-driver-node-hqvb4\" (UID: \"caba445d-47e5-446e-9be4-9148e4d7e59a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqvb4"
Jul 28 22:05:04.978553 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.978257    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2a9cac08-b862-4f7b-88e3-f4de98a035ea-serviceca\") pod \"node-ca-tx5m6\" (UID: \"2a9cac08-b862-4f7b-88e3-f4de98a035ea\") " pod="openshift-image-registry/node-ca-tx5m6"
Jul 28 22:05:04.978553 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.978283    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-host-var-lib-cni-bin\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:04.978553 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.978305    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-hostroot\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:04.978553 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.978381    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d7dfdb2-a77a-47e8-865a-07a014ec5633-original-pull-secret\") pod \"global-pull-secret-syncer-2rqhg\" (UID: \"5d7dfdb2-a77a-47e8-865a-07a014ec5633\") " pod="kube-system/global-pull-secret-syncer-2rqhg"
Jul 28 22:05:04.978553 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.978399    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 22:05:04.978553 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.978418    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/caba445d-47e5-446e-9be4-9148e4d7e59a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hqvb4\" (UID: \"caba445d-47e5-446e-9be4-9148e4d7e59a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqvb4"
Jul 28 22:05:04.978553 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.978451    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6eea54c2-c579-4dab-8222-909d88ccf205-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4fjz6\" (UID: \"6eea54c2-c579-4dab-8222-909d88ccf205\") " pod="openshift-multus/multus-additional-cni-plugins-4fjz6"
Jul 28 22:05:04.978553 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.978485    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-multus-cni-dir\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:04.978553 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.978511    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-multus-conf-dir\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:04.978553 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.978547    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8dsx\" (UniqueName: \"kubernetes.io/projected/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-kube-api-access-g8dsx\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:04.979128 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.978569    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2a9cac08-b862-4f7b-88e3-f4de98a035ea-host\") pod \"node-ca-tx5m6\" (UID: \"2a9cac08-b862-4f7b-88e3-f4de98a035ea\") " pod="openshift-image-registry/node-ca-tx5m6"
Jul 28 22:05:04.979128 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.978586    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6eea54c2-c579-4dab-8222-909d88ccf205-cni-binary-copy\") pod \"multus-additional-cni-plugins-4fjz6\" (UID: \"6eea54c2-c579-4dab-8222-909d88ccf205\") " pod="openshift-multus/multus-additional-cni-plugins-4fjz6"
Jul 28 22:05:04.979128 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.978629    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 22:05:04.979128 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.978634    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6eea54c2-c579-4dab-8222-909d88ccf205-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4fjz6\" (UID: \"6eea54c2-c579-4dab-8222-909d88ccf205\") " pod="openshift-multus/multus-additional-cni-plugins-4fjz6"
Jul 28 22:05:04.979128 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.978652    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-system-cni-dir\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:04.979128 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.978667    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-cnibin\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:04.979128 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.978688    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-host-var-lib-cni-multus\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:04.979128 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.978721    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-87zr2"
Jul 28 22:05:04.979128 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.978727    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-multus-daemon-config\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:04.979128 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.978771    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/820ee419-dab7-43ef-94bf-cadbaf9a5232-hosts-file\") pod \"node-resolver-qtgkr\" (UID: \"820ee419-dab7-43ef-94bf-cadbaf9a5232\") " pod="openshift-dns/node-resolver-qtgkr"
Jul 28 22:05:04.979128 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.978801    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/caba445d-47e5-446e-9be4-9148e4d7e59a-socket-dir\") pod \"aws-ebs-csi-driver-node-hqvb4\" (UID: \"caba445d-47e5-446e-9be4-9148e4d7e59a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqvb4"
Jul 28 22:05:04.979128 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:04.978801    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-87zr2" podUID="962b24a3-abd7-4a45-98fa-f225c383b0dc"
Jul 28 22:05:04.979128 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.978833    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/caba445d-47e5-446e-9be4-9148e4d7e59a-device-dir\") pod \"aws-ebs-csi-driver-node-hqvb4\" (UID: \"caba445d-47e5-446e-9be4-9148e4d7e59a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqvb4"
Jul 28 22:05:04.979128 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.978872    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6eea54c2-c579-4dab-8222-909d88ccf205-os-release\") pod \"multus-additional-cni-plugins-4fjz6\" (UID: \"6eea54c2-c579-4dab-8222-909d88ccf205\") " pod="openshift-multus/multus-additional-cni-plugins-4fjz6"
Jul 28 22:05:04.979128 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.978891    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-host-run-netns\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:04.979128 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.978908    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-etc-kubernetes\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:04.979128 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.978946    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjh26\" (UniqueName: \"kubernetes.io/projected/caba445d-47e5-446e-9be4-9148e4d7e59a-kube-api-access-tjh26\") pod \"aws-ebs-csi-driver-node-hqvb4\" (UID: \"caba445d-47e5-446e-9be4-9148e4d7e59a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqvb4"
Jul 28 22:05:04.979763 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.978963    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6eea54c2-c579-4dab-8222-909d88ccf205-system-cni-dir\") pod \"multus-additional-cni-plugins-4fjz6\" (UID: \"6eea54c2-c579-4dab-8222-909d88ccf205\") " pod="openshift-multus/multus-additional-cni-plugins-4fjz6"
Jul 28 22:05:04.979763 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.978983    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-host-run-multus-certs\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:04.979763 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.979018    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/820ee419-dab7-43ef-94bf-cadbaf9a5232-tmp-dir\") pod \"node-resolver-qtgkr\" (UID: \"820ee419-dab7-43ef-94bf-cadbaf9a5232\") " pod="openshift-dns/node-resolver-qtgkr"
Jul 28 22:05:04.979763 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.979051    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvcx9\" (UniqueName: \"kubernetes.io/projected/2a9cac08-b862-4f7b-88e3-f4de98a035ea-kube-api-access-vvcx9\") pod \"node-ca-tx5m6\" (UID: \"2a9cac08-b862-4f7b-88e3-f4de98a035ea\") " pod="openshift-image-registry/node-ca-tx5m6"
Jul 28 22:05:04.979763 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.979083    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6eea54c2-c579-4dab-8222-909d88ccf205-cnibin\") pod \"multus-additional-cni-plugins-4fjz6\" (UID: \"6eea54c2-c579-4dab-8222-909d88ccf205\") " pod="openshift-multus/multus-additional-cni-plugins-4fjz6"
Jul 28 22:05:04.979763 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.979129    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f2vm\" (UniqueName: \"kubernetes.io/projected/6eea54c2-c579-4dab-8222-909d88ccf205-kube-api-access-4f2vm\") pod \"multus-additional-cni-plugins-4fjz6\" (UID: \"6eea54c2-c579-4dab-8222-909d88ccf205\") " pod="openshift-multus/multus-additional-cni-plugins-4fjz6"
Jul 28 22:05:04.979763 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.979165    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5d7dfdb2-a77a-47e8-865a-07a014ec5633-dbus\") pod \"global-pull-secret-syncer-2rqhg\" (UID: \"5d7dfdb2-a77a-47e8-865a-07a014ec5633\") " pod="kube-system/global-pull-secret-syncer-2rqhg"
Jul 28 22:05:04.979763 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.979207    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/caba445d-47e5-446e-9be4-9148e4d7e59a-sys-fs\") pod \"aws-ebs-csi-driver-node-hqvb4\" (UID: \"caba445d-47e5-446e-9be4-9148e4d7e59a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqvb4"
Jul 28 22:05:04.979763 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.979232    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6eea54c2-c579-4dab-8222-909d88ccf205-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4fjz6\" (UID: \"6eea54c2-c579-4dab-8222-909d88ccf205\") " pod="openshift-multus/multus-additional-cni-plugins-4fjz6"
Jul 28 22:05:04.979763 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.979260    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-cni-binary-copy\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:04.979763 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.979288    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-multus-socket-dir-parent\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:04.979763 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.979312    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-os-release\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:04.979763 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.979337    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-host-run-k8s-cni-cncf-io\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:04.979763 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.979360    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-host-var-lib-kubelet\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:04.980565 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.980536    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:04.980644 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.980591    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 22:05:04.980644 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.980605    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 22:05:04.980644 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.980622    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xff8v\""
Jul 28 22:05:04.980867 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.980654    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 22:05:04.980867 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.980653    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jlvdh\""
Jul 28 22:05:04.980968 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.980864    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 22:05:04.981248 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.981234    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 22:05:04.983928 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.983908    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 22:05:04.984214 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.984197    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 22:05:04.984299 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:04.984284    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pwb2t\""
Jul 28 22:05:05.008198 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.008176    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ssj22"
Jul 28 22:05:05.021608 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.021591    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ssj22"
Jul 28 22:05:05.071208 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.070373    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 22:05:05.079635 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.079606    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6eea54c2-c579-4dab-8222-909d88ccf205-system-cni-dir\") pod \"multus-additional-cni-plugins-4fjz6\" (UID: \"6eea54c2-c579-4dab-8222-909d88ccf205\") " pod="openshift-multus/multus-additional-cni-plugins-4fjz6"
Jul 28 22:05:05.079743 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.079645    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-host-run-multus-certs\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:05.079743 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.079681    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8347000f-940f-497d-af0c-71f180037642-iptables-alerter-script\") pod \"iptables-alerter-pwnwp\" (UID: \"8347000f-940f-497d-af0c-71f180037642\") " pod="openshift-network-operator/iptables-alerter-pwnwp"
Jul 28 22:05:05.079838 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.079747    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6eea54c2-c579-4dab-8222-909d88ccf205-system-cni-dir\") pod \"multus-additional-cni-plugins-4fjz6\" (UID: \"6eea54c2-c579-4dab-8222-909d88ccf205\") " pod="openshift-multus/multus-additional-cni-plugins-4fjz6"
Jul 28 22:05:05.079838 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.079776    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/820ee419-dab7-43ef-94bf-cadbaf9a5232-tmp-dir\") pod \"node-resolver-qtgkr\" (UID: \"820ee419-dab7-43ef-94bf-cadbaf9a5232\") " pod="openshift-dns/node-resolver-qtgkr"
Jul 28 22:05:05.079838 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.079810    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvcx9\" (UniqueName: \"kubernetes.io/projected/2a9cac08-b862-4f7b-88e3-f4de98a035ea-kube-api-access-vvcx9\") pod \"node-ca-tx5m6\" (UID: \"2a9cac08-b862-4f7b-88e3-f4de98a035ea\") " pod="openshift-image-registry/node-ca-tx5m6"
Jul 28 22:05:05.080009 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.079751    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-host-run-multus-certs\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:05.080009 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.079904    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f2vm\" (UniqueName: \"kubernetes.io/projected/6eea54c2-c579-4dab-8222-909d88ccf205-kube-api-access-4f2vm\") pod \"multus-additional-cni-plugins-4fjz6\" (UID: \"6eea54c2-c579-4dab-8222-909d88ccf205\") " pod="openshift-multus/multus-additional-cni-plugins-4fjz6"
Jul 28 22:05:05.080009 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.079944    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-host-kubelet\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.080009 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.079972    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-host-run-netns\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.080009 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.080000    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-node-log\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.080196 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.080024    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-host-run-ovn-kubernetes\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.080196 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.080048    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7bab32b0-13c6-4055-916e-a7454bafb5b6-ovn-node-metrics-cert\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.080196 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.080079    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5d7dfdb2-a77a-47e8-865a-07a014ec5633-dbus\") pod \"global-pull-secret-syncer-2rqhg\" (UID: \"5d7dfdb2-a77a-47e8-865a-07a014ec5633\") " pod="kube-system/global-pull-secret-syncer-2rqhg"
Jul 28 22:05:05.080196 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.080114    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-multus-socket-dir-parent\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:05.080196 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.080146    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7bab32b0-13c6-4055-916e-a7454bafb5b6-env-overrides\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.080196 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.080169    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcf7b9b9-4c50-4c88-8190-4f62618f849d-etc-kubernetes\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.080196 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.080191    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bcf7b9b9-4c50-4c88-8190-4f62618f849d-run\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.080516 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.080196    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/820ee419-dab7-43ef-94bf-cadbaf9a5232-tmp-dir\") pod \"node-resolver-qtgkr\" (UID: \"820ee419-dab7-43ef-94bf-cadbaf9a5232\") " pod="openshift-dns/node-resolver-qtgkr"
Jul 28 22:05:05.080516 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.080228    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-os-release\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:05.080516 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.080259    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-host-run-k8s-cni-cncf-io\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:05.080516 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.080292    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-host-var-lib-kubelet\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:05.080516 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.080312    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-os-release\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:05.080516 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.080317    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-host-cni-netd\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.080516 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.080316    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-multus-socket-dir-parent\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:05.080516 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.080318    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5d7dfdb2-a77a-47e8-865a-07a014ec5633-dbus\") pod \"global-pull-secret-syncer-2rqhg\" (UID: \"5d7dfdb2-a77a-47e8-865a-07a014ec5633\") " pod="kube-system/global-pull-secret-syncer-2rqhg"
Jul 28 22:05:05.080516 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.080352    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7bab32b0-13c6-4055-916e-a7454bafb5b6-ovnkube-script-lib\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.080516 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.080355    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-host-var-lib-kubelet\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:05.080516 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.080402    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-etc-openvswitch\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.080516 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.080402    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-host-run-k8s-cni-cncf-io\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:05.080516 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.080433    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/caba445d-47e5-446e-9be4-9148e4d7e59a-etc-selinux\") pod \"aws-ebs-csi-driver-node-hqvb4\" (UID: \"caba445d-47e5-446e-9be4-9148e4d7e59a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqvb4"
Jul 28 22:05:05.080516 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.080454    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2a9cac08-b862-4f7b-88e3-f4de98a035ea-serviceca\") pod \"node-ca-tx5m6\" (UID: \"2a9cac08-b862-4f7b-88e3-f4de98a035ea\") " pod="openshift-image-registry/node-ca-tx5m6"
Jul 28 22:05:05.080516 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.080475    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-host-var-lib-cni-bin\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:05.080516 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.080494    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-hostroot\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:05.080516 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.080520    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/caba445d-47e5-446e-9be4-9148e4d7e59a-etc-selinux\") pod \"aws-ebs-csi-driver-node-hqvb4\" (UID: \"caba445d-47e5-446e-9be4-9148e4d7e59a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqvb4"
Jul 28 22:05:05.081347 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.080532    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-hostroot\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:05.081347 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.080553    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-host-var-lib-cni-bin\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:05.081347 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.080606    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8347000f-940f-497d-af0c-71f180037642-host-slash\") pod \"iptables-alerter-pwnwp\" (UID: \"8347000f-940f-497d-af0c-71f180037642\") " pod="openshift-network-operator/iptables-alerter-pwnwp"
Jul 28 22:05:05.081347 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.080634    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f33e531-3341-4c01-940e-91f17519976c-metrics-certs\") pod \"network-metrics-daemon-4xqhf\" (UID: \"2f33e531-3341-4c01-940e-91f17519976c\") " pod="openshift-multus/network-metrics-daemon-4xqhf"
Jul 28 22:05:05.081347 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.080659    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-run-openvswitch\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.081347 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.080707    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-systemd-units\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.081347 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.080737    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/caba445d-47e5-446e-9be4-9148e4d7e59a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hqvb4\" (UID: \"caba445d-47e5-446e-9be4-9148e4d7e59a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqvb4"
Jul 28 22:05:05.081347 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.080795    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/caba445d-47e5-446e-9be4-9148e4d7e59a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hqvb4\" (UID: \"caba445d-47e5-446e-9be4-9148e4d7e59a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqvb4"
Jul 28 22:05:05.081347 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.080802    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6eea54c2-c579-4dab-8222-909d88ccf205-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4fjz6\" (UID: \"6eea54c2-c579-4dab-8222-909d88ccf205\") " pod="openshift-multus/multus-additional-cni-plugins-4fjz6"
Jul 28 22:05:05.081347 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.080823    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8dsx\" (UniqueName: \"kubernetes.io/projected/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-kube-api-access-g8dsx\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:05.081347 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.080868    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gvhx\" (UniqueName: \"kubernetes.io/projected/2f33e531-3341-4c01-940e-91f17519976c-kube-api-access-8gvhx\") pod \"network-metrics-daemon-4xqhf\" (UID: \"2f33e531-3341-4c01-940e-91f17519976c\") " pod="openshift-multus/network-metrics-daemon-4xqhf"
Jul 28 22:05:05.081347 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.080902    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7bab32b0-13c6-4055-916e-a7454bafb5b6-ovnkube-config\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.081347 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.080959    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bcf7b9b9-4c50-4c88-8190-4f62618f849d-etc-sysconfig\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.081347 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.080988    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2a9cac08-b862-4f7b-88e3-f4de98a035ea-serviceca\") pod \"node-ca-tx5m6\" (UID: \"2a9cac08-b862-4f7b-88e3-f4de98a035ea\") " pod="openshift-image-registry/node-ca-tx5m6"
Jul 28 22:05:05.081347 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.080982    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bcf7b9b9-4c50-4c88-8190-4f62618f849d-sys\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.081347 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.081056    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-etc-kubernetes\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:05.081347 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.081088    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcf7b9b9-4c50-4c88-8190-4f62618f849d-var-lib-kubelet\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.082191 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.081092    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-etc-kubernetes\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:05.082191 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.081133    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6eea54c2-c579-4dab-8222-909d88ccf205-os-release\") pod \"multus-additional-cni-plugins-4fjz6\" (UID: \"6eea54c2-c579-4dab-8222-909d88ccf205\") " pod="openshift-multus/multus-additional-cni-plugins-4fjz6"
Jul 28 22:05:05.082191 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.081159    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-host-cni-bin\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.082191 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.081184    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6f95fc9f-b8c8-433e-ae9b-e20fcd167fbe-konnectivity-ca\") pod \"konnectivity-agent-7wmm5\" (UID: \"6f95fc9f-b8c8-433e-ae9b-e20fcd167fbe\") " pod="kube-system/konnectivity-agent-7wmm5"
Jul 28 22:05:05.082191 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.081226    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bcf7b9b9-4c50-4c88-8190-4f62618f849d-etc-sysctl-conf\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.082191 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.081234    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6eea54c2-c579-4dab-8222-909d88ccf205-os-release\") pod \"multus-additional-cni-plugins-4fjz6\" (UID: \"6eea54c2-c579-4dab-8222-909d88ccf205\") " pod="openshift-multus/multus-additional-cni-plugins-4fjz6"
Jul 28 22:05:05.082191 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.081247    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bcf7b9b9-4c50-4c88-8190-4f62618f849d-etc-systemd\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.082191 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.081284    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjh26\" (UniqueName: \"kubernetes.io/projected/caba445d-47e5-446e-9be4-9148e4d7e59a-kube-api-access-tjh26\") pod \"aws-ebs-csi-driver-node-hqvb4\" (UID: \"caba445d-47e5-446e-9be4-9148e4d7e59a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqvb4"
Jul 28 22:05:05.082191 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.081315    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6f95fc9f-b8c8-433e-ae9b-e20fcd167fbe-agent-certs\") pod \"konnectivity-agent-7wmm5\" (UID: \"6f95fc9f-b8c8-433e-ae9b-e20fcd167fbe\") " pod="kube-system/konnectivity-agent-7wmm5"
Jul 28 22:05:05.082191 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.081316    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6eea54c2-c579-4dab-8222-909d88ccf205-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4fjz6\" (UID: \"6eea54c2-c579-4dab-8222-909d88ccf205\") " pod="openshift-multus/multus-additional-cni-plugins-4fjz6"
Jul 28 22:05:05.082191 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.081342    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6eea54c2-c579-4dab-8222-909d88ccf205-cnibin\") pod \"multus-additional-cni-plugins-4fjz6\" (UID: \"6eea54c2-c579-4dab-8222-909d88ccf205\") " pod="openshift-multus/multus-additional-cni-plugins-4fjz6"
Jul 28 22:05:05.082191 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.081389    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-var-lib-openvswitch\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.082191 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.081437    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-run-ovn\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.082191 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.081462    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bcf7b9b9-4c50-4c88-8190-4f62618f849d-etc-tuned\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.082191 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.081490    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6eea54c2-c579-4dab-8222-909d88ccf205-cnibin\") pod \"multus-additional-cni-plugins-4fjz6\" (UID: \"6eea54c2-c579-4dab-8222-909d88ccf205\") " pod="openshift-multus/multus-additional-cni-plugins-4fjz6"
Jul 28 22:05:05.082191 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.081499    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/caba445d-47e5-446e-9be4-9148e4d7e59a-sys-fs\") pod \"aws-ebs-csi-driver-node-hqvb4\" (UID: \"caba445d-47e5-446e-9be4-9148e4d7e59a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqvb4"
Jul 28 22:05:05.082191 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.081554    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6eea54c2-c579-4dab-8222-909d88ccf205-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4fjz6\" (UID: \"6eea54c2-c579-4dab-8222-909d88ccf205\") " pod="openshift-multus/multus-additional-cni-plugins-4fjz6"
Jul 28 22:05:05.083079 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.081562    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/caba445d-47e5-446e-9be4-9148e4d7e59a-sys-fs\") pod \"aws-ebs-csi-driver-node-hqvb4\" (UID: \"caba445d-47e5-446e-9be4-9148e4d7e59a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqvb4"
Jul 28 22:05:05.083079 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.081580    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-cni-binary-copy\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:05.083079 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.081599    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-run-systemd\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.083079 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.081626    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bcf7b9b9-4c50-4c88-8190-4f62618f849d-host\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.083079 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.081697    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh5wd\" (UniqueName: \"kubernetes.io/projected/8347000f-940f-497d-af0c-71f180037642-kube-api-access-jh5wd\") pod \"iptables-alerter-pwnwp\" (UID: \"8347000f-940f-497d-af0c-71f180037642\") " pod="openshift-network-operator/iptables-alerter-pwnwp"
Jul 28 22:05:05.083079 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.081750    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-host-slash\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.083079 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.081788    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zzcp\" (UniqueName: \"kubernetes.io/projected/7bab32b0-13c6-4055-916e-a7454bafb5b6-kube-api-access-7zzcp\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.083079 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.081817    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52t5j\" (UniqueName: \"kubernetes.io/projected/820ee419-dab7-43ef-94bf-cadbaf9a5232-kube-api-access-52t5j\") pod \"node-resolver-qtgkr\" (UID: \"820ee419-dab7-43ef-94bf-cadbaf9a5232\") " pod="openshift-dns/node-resolver-qtgkr"
Jul 28 22:05:05.083079 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.081843    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5d7dfdb2-a77a-47e8-865a-07a014ec5633-kubelet-config\") pod \"global-pull-secret-syncer-2rqhg\" (UID: \"5d7dfdb2-a77a-47e8-865a-07a014ec5633\") " pod="kube-system/global-pull-secret-syncer-2rqhg"
Jul 28 22:05:05.083079 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.081864    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/caba445d-47e5-446e-9be4-9148e4d7e59a-registration-dir\") pod \"aws-ebs-csi-driver-node-hqvb4\" (UID: \"caba445d-47e5-446e-9be4-9148e4d7e59a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqvb4"
Jul 28 22:05:05.083079 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.081879    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bcf7b9b9-4c50-4c88-8190-4f62618f849d-etc-modprobe-d\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.083079 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.081899    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcf7b9b9-4c50-4c88-8190-4f62618f849d-tmp\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.083079 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.081923    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwqhl\" (UniqueName: \"kubernetes.io/projected/bcf7b9b9-4c50-4c88-8190-4f62618f849d-kube-api-access-cwqhl\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.083079 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.081949    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d7dfdb2-a77a-47e8-865a-07a014ec5633-original-pull-secret\") pod \"global-pull-secret-syncer-2rqhg\" (UID: \"5d7dfdb2-a77a-47e8-865a-07a014ec5633\") " pod="kube-system/global-pull-secret-syncer-2rqhg"
Jul 28 22:05:05.083079 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.081953    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5d7dfdb2-a77a-47e8-865a-07a014ec5633-kubelet-config\") pod \"global-pull-secret-syncer-2rqhg\" (UID: \"5d7dfdb2-a77a-47e8-865a-07a014ec5633\") " pod="kube-system/global-pull-secret-syncer-2rqhg"
Jul 28 22:05:05.083079 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.082009    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-multus-cni-dir\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:05.083079 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:05.082041    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 22:05:05.083736 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.082050    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-multus-conf-dir\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:05.083736 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.082106    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntdgg\" (UniqueName: \"kubernetes.io/projected/962b24a3-abd7-4a45-98fa-f225c383b0dc-kube-api-access-ntdgg\") pod \"network-check-target-87zr2\" (UID: \"962b24a3-abd7-4a45-98fa-f225c383b0dc\") " pod="openshift-network-diagnostics/network-check-target-87zr2"
Jul 28 22:05:05.083736 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.082130    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bcf7b9b9-4c50-4c88-8190-4f62618f849d-lib-modules\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.083736 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.082156    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6eea54c2-c579-4dab-8222-909d88ccf205-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4fjz6\" (UID: \"6eea54c2-c579-4dab-8222-909d88ccf205\") " pod="openshift-multus/multus-additional-cni-plugins-4fjz6"
Jul 28 22:05:05.083736 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.082179    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2a9cac08-b862-4f7b-88e3-f4de98a035ea-host\") pod \"node-ca-tx5m6\" (UID: \"2a9cac08-b862-4f7b-88e3-f4de98a035ea\") " pod="openshift-image-registry/node-ca-tx5m6"
Jul 28 22:05:05.083736 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:05.082242    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d7dfdb2-a77a-47e8-865a-07a014ec5633-original-pull-secret podName:5d7dfdb2-a77a-47e8-865a-07a014ec5633 nodeName:}" failed. No retries permitted until 2026-07-28 22:05:05.582184108 +0000 UTC m=+2.962321738 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d7dfdb2-a77a-47e8-865a-07a014ec5633-original-pull-secret") pod "global-pull-secret-syncer-2rqhg" (UID: "5d7dfdb2-a77a-47e8-865a-07a014ec5633") : object "kube-system"/"original-pull-secret" not registered
Jul 28 22:05:05.083736 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.082246    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-multus-conf-dir\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:05.083736 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.082266    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-cni-binary-copy\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:05.083736 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.082223    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2a9cac08-b862-4f7b-88e3-f4de98a035ea-host\") pod \"node-ca-tx5m6\" (UID: \"2a9cac08-b862-4f7b-88e3-f4de98a035ea\") " pod="openshift-image-registry/node-ca-tx5m6"
Jul 28 22:05:05.083736 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.082310    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6eea54c2-c579-4dab-8222-909d88ccf205-cni-binary-copy\") pod \"multus-additional-cni-plugins-4fjz6\" (UID: \"6eea54c2-c579-4dab-8222-909d88ccf205\") " pod="openshift-multus/multus-additional-cni-plugins-4fjz6"
Jul 28 22:05:05.083736 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.082343    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6eea54c2-c579-4dab-8222-909d88ccf205-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4fjz6\" (UID: \"6eea54c2-c579-4dab-8222-909d88ccf205\") " pod="openshift-multus/multus-additional-cni-plugins-4fjz6"
Jul 28 22:05:05.083736 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.082370    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-system-cni-dir\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:05.083736 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.082380    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-multus-cni-dir\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:05.083736 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.082398    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/caba445d-47e5-446e-9be4-9148e4d7e59a-registration-dir\") pod \"aws-ebs-csi-driver-node-hqvb4\" (UID: \"caba445d-47e5-446e-9be4-9148e4d7e59a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqvb4"
Jul 28 22:05:05.083736 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.082420    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-cnibin\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:05.083736 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.082448    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-host-var-lib-cni-multus\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:05.083736 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.082447    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-system-cni-dir\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:05.084318 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.082474    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-multus-daemon-config\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:05.084318 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.082499    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/820ee419-dab7-43ef-94bf-cadbaf9a5232-hosts-file\") pod \"node-resolver-qtgkr\" (UID: \"820ee419-dab7-43ef-94bf-cadbaf9a5232\") " pod="openshift-dns/node-resolver-qtgkr"
Jul 28 22:05:05.084318 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.082506    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-host-var-lib-cni-multus\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:05.084318 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.082511    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6eea54c2-c579-4dab-8222-909d88ccf205-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4fjz6\" (UID: \"6eea54c2-c579-4dab-8222-909d88ccf205\") " pod="openshift-multus/multus-additional-cni-plugins-4fjz6"
Jul 28 22:05:05.084318 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.082519    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-cnibin\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:05.084318 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.082523    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/caba445d-47e5-446e-9be4-9148e4d7e59a-socket-dir\") pod \"aws-ebs-csi-driver-node-hqvb4\" (UID: \"caba445d-47e5-446e-9be4-9148e4d7e59a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqvb4"
Jul 28 22:05:05.084318 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.082582    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/820ee419-dab7-43ef-94bf-cadbaf9a5232-hosts-file\") pod \"node-resolver-qtgkr\" (UID: \"820ee419-dab7-43ef-94bf-cadbaf9a5232\") " pod="openshift-dns/node-resolver-qtgkr"
Jul 28 22:05:05.084318 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.082615    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/caba445d-47e5-446e-9be4-9148e4d7e59a-device-dir\") pod \"aws-ebs-csi-driver-node-hqvb4\" (UID: \"caba445d-47e5-446e-9be4-9148e4d7e59a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqvb4"
Jul 28 22:05:05.084318 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.082642    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/caba445d-47e5-446e-9be4-9148e4d7e59a-socket-dir\") pod \"aws-ebs-csi-driver-node-hqvb4\" (UID: \"caba445d-47e5-446e-9be4-9148e4d7e59a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqvb4"
Jul 28 22:05:05.084318 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.082649    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-host-run-netns\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:05.084318 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.082664    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/caba445d-47e5-446e-9be4-9148e4d7e59a-device-dir\") pod \"aws-ebs-csi-driver-node-hqvb4\" (UID: \"caba445d-47e5-446e-9be4-9148e4d7e59a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqvb4"
Jul 28 22:05:05.084318 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.082677    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-log-socket\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.084318 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.082718    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-host-run-netns\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:05.084318 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.082757    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.084318 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.082783    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bcf7b9b9-4c50-4c88-8190-4f62618f849d-etc-sysctl-d\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.084318 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.082831    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6eea54c2-c579-4dab-8222-909d88ccf205-cni-binary-copy\") pod \"multus-additional-cni-plugins-4fjz6\" (UID: \"6eea54c2-c579-4dab-8222-909d88ccf205\") " pod="openshift-multus/multus-additional-cni-plugins-4fjz6"
Jul 28 22:05:05.084318 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.083030    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-multus-daemon-config\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:05.087690 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.087652    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 22:05:05.091875 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.091843    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjh26\" (UniqueName: \"kubernetes.io/projected/caba445d-47e5-446e-9be4-9148e4d7e59a-kube-api-access-tjh26\") pod \"aws-ebs-csi-driver-node-hqvb4\" (UID: \"caba445d-47e5-446e-9be4-9148e4d7e59a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqvb4"
Jul 28 22:05:05.091981 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.091878    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f2vm\" (UniqueName: \"kubernetes.io/projected/6eea54c2-c579-4dab-8222-909d88ccf205-kube-api-access-4f2vm\") pod \"multus-additional-cni-plugins-4fjz6\" (UID: \"6eea54c2-c579-4dab-8222-909d88ccf205\") " pod="openshift-multus/multus-additional-cni-plugins-4fjz6"
Jul 28 22:05:05.091981 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.091888    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvcx9\" (UniqueName: \"kubernetes.io/projected/2a9cac08-b862-4f7b-88e3-f4de98a035ea-kube-api-access-vvcx9\") pod \"node-ca-tx5m6\" (UID: \"2a9cac08-b862-4f7b-88e3-f4de98a035ea\") " pod="openshift-image-registry/node-ca-tx5m6"
Jul 28 22:05:05.091981 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.091885    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8dsx\" (UniqueName: \"kubernetes.io/projected/1bd94e5c-7c0a-47d0-8de0-9ac456d7f658-kube-api-access-g8dsx\") pod \"multus-pkhlk\" (UID: \"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658\") " pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:05.092291 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.092269    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52t5j\" (UniqueName: \"kubernetes.io/projected/820ee419-dab7-43ef-94bf-cadbaf9a5232-kube-api-access-52t5j\") pod \"node-resolver-qtgkr\" (UID: \"820ee419-dab7-43ef-94bf-cadbaf9a5232\") " pod="openshift-dns/node-resolver-qtgkr"
Jul 28 22:05:05.124692 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.124667    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 22:05:05.130320 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.130230    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal" event={"ID":"e5ce7b7b59f5b560ebbfcc3f6f472e5b","Type":"ContainerStarted","Data":"40e3ac3bc03e0ef16f82ff4bfb020216d2652df4a1bcebdfbd241d90d0ea9f92"}
Jul 28 22:05:05.132419 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.132352    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-37.ec2.internal" event={"ID":"2aaf76ae458a97a3e5edaa3e36196e52","Type":"ContainerStarted","Data":"727d8cc58ec803ee8420b78918afdb2570a7f44bcbacba55a631d44743c8b0e8"}
Jul 28 22:05:05.183643 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.183605    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-host-cni-bin\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.183643 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.183647    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6f95fc9f-b8c8-433e-ae9b-e20fcd167fbe-konnectivity-ca\") pod \"konnectivity-agent-7wmm5\" (UID: \"6f95fc9f-b8c8-433e-ae9b-e20fcd167fbe\") " pod="kube-system/konnectivity-agent-7wmm5"
Jul 28 22:05:05.183949 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.183671    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bcf7b9b9-4c50-4c88-8190-4f62618f849d-etc-sysctl-conf\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.183949 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.183735    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-host-cni-bin\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.183949 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.183774    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bcf7b9b9-4c50-4c88-8190-4f62618f849d-etc-systemd\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.183949 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.183798    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6f95fc9f-b8c8-433e-ae9b-e20fcd167fbe-agent-certs\") pod \"konnectivity-agent-7wmm5\" (UID: \"6f95fc9f-b8c8-433e-ae9b-e20fcd167fbe\") " pod="kube-system/konnectivity-agent-7wmm5"
Jul 28 22:05:05.183949 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.183826    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bcf7b9b9-4c50-4c88-8190-4f62618f849d-etc-sysctl-conf\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.183949 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.183828    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-var-lib-openvswitch\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.183949 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.183863    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-var-lib-openvswitch\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.183949 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.183868    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-run-ovn\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.183949 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.183894    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bcf7b9b9-4c50-4c88-8190-4f62618f849d-etc-tuned\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.183949 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.183910    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bcf7b9b9-4c50-4c88-8190-4f62618f849d-etc-systemd\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.183949 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.183923    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-run-systemd\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.183949 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.183951    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-run-ovn\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.185858 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.183951    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bcf7b9b9-4c50-4c88-8190-4f62618f849d-host\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.185858 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.183988    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh5wd\" (UniqueName: \"kubernetes.io/projected/8347000f-940f-497d-af0c-71f180037642-kube-api-access-jh5wd\") pod \"iptables-alerter-pwnwp\" (UID: \"8347000f-940f-497d-af0c-71f180037642\") " pod="openshift-network-operator/iptables-alerter-pwnwp"
Jul 28 22:05:05.185858 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.183997    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bcf7b9b9-4c50-4c88-8190-4f62618f849d-host\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.185858 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.184014    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-host-slash\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.185858 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.184075    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zzcp\" (UniqueName: \"kubernetes.io/projected/7bab32b0-13c6-4055-916e-a7454bafb5b6-kube-api-access-7zzcp\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.185858 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.184105    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bcf7b9b9-4c50-4c88-8190-4f62618f849d-etc-modprobe-d\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.185858 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.184130    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcf7b9b9-4c50-4c88-8190-4f62618f849d-tmp\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.185858 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.184155    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwqhl\" (UniqueName: \"kubernetes.io/projected/bcf7b9b9-4c50-4c88-8190-4f62618f849d-kube-api-access-cwqhl\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.185858 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.184199    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntdgg\" (UniqueName: \"kubernetes.io/projected/962b24a3-abd7-4a45-98fa-f225c383b0dc-kube-api-access-ntdgg\") pod \"network-check-target-87zr2\" (UID: \"962b24a3-abd7-4a45-98fa-f225c383b0dc\") " pod="openshift-network-diagnostics/network-check-target-87zr2"
Jul 28 22:05:05.185858 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.184225    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bcf7b9b9-4c50-4c88-8190-4f62618f849d-lib-modules\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.185858 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.184261    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-log-socket\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.185858 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.184291    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.185858 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.184299    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6f95fc9f-b8c8-433e-ae9b-e20fcd167fbe-konnectivity-ca\") pod \"konnectivity-agent-7wmm5\" (UID: \"6f95fc9f-b8c8-433e-ae9b-e20fcd167fbe\") " pod="kube-system/konnectivity-agent-7wmm5"
Jul 28 22:05:05.185858 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.184319    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bcf7b9b9-4c50-4c88-8190-4f62618f849d-etc-sysctl-d\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.185858 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.184368    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8347000f-940f-497d-af0c-71f180037642-iptables-alerter-script\") pod \"iptables-alerter-pwnwp\" (UID: \"8347000f-940f-497d-af0c-71f180037642\") " pod="openshift-network-operator/iptables-alerter-pwnwp"
Jul 28 22:05:05.185858 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.184428    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-host-kubelet\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.185858 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.184497    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-log-socket\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.186788 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.184526    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bcf7b9b9-4c50-4c88-8190-4f62618f849d-lib-modules\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.186788 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.184556    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-host-kubelet\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.186788 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.184622    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bcf7b9b9-4c50-4c88-8190-4f62618f849d-etc-sysctl-d\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.186788 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.184635    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.186788 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.184665    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-host-run-netns\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.186788 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.184698    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-node-log\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.186788 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.184718    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-run-systemd\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.186788 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.184725    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-host-run-ovn-kubernetes\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.186788 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.184761    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-host-run-netns\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.186788 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.184768    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bcf7b9b9-4c50-4c88-8190-4f62618f849d-etc-modprobe-d\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.186788 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.184754    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7bab32b0-13c6-4055-916e-a7454bafb5b6-ovn-node-metrics-cert\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.186788 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.184784    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-node-log\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.186788 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.184934    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-host-run-ovn-kubernetes\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.186788 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.185028    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7bab32b0-13c6-4055-916e-a7454bafb5b6-env-overrides\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.186788 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.185064    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8347000f-940f-497d-af0c-71f180037642-iptables-alerter-script\") pod \"iptables-alerter-pwnwp\" (UID: \"8347000f-940f-497d-af0c-71f180037642\") " pod="openshift-network-operator/iptables-alerter-pwnwp"
Jul 28 22:05:05.186788 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.185089    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcf7b9b9-4c50-4c88-8190-4f62618f849d-etc-kubernetes\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.186788 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.185117    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bcf7b9b9-4c50-4c88-8190-4f62618f849d-run\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.187682 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.185147    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-host-cni-netd\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.187682 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.185174    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7bab32b0-13c6-4055-916e-a7454bafb5b6-ovnkube-script-lib\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.187682 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.185201    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-etc-openvswitch\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.187682 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.185215    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bcf7b9b9-4c50-4c88-8190-4f62618f849d-run\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.187682 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.185231    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8347000f-940f-497d-af0c-71f180037642-host-slash\") pod \"iptables-alerter-pwnwp\" (UID: \"8347000f-940f-497d-af0c-71f180037642\") " pod="openshift-network-operator/iptables-alerter-pwnwp"
Jul 28 22:05:05.187682 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.185255    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-host-cni-netd\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.187682 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.185274    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcf7b9b9-4c50-4c88-8190-4f62618f849d-etc-kubernetes\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.187682 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.185257    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f33e531-3341-4c01-940e-91f17519976c-metrics-certs\") pod \"network-metrics-daemon-4xqhf\" (UID: \"2f33e531-3341-4c01-940e-91f17519976c\") " pod="openshift-multus/network-metrics-daemon-4xqhf"
Jul 28 22:05:05.187682 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.185315    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-etc-openvswitch\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.187682 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.185329    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-run-openvswitch\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.187682 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:05.185337    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 22:05:05.187682 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.185359    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-systemd-units\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.187682 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.185395    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gvhx\" (UniqueName: \"kubernetes.io/projected/2f33e531-3341-4c01-940e-91f17519976c-kube-api-access-8gvhx\") pod \"network-metrics-daemon-4xqhf\" (UID: \"2f33e531-3341-4c01-940e-91f17519976c\") " pod="openshift-multus/network-metrics-daemon-4xqhf"
Jul 28 22:05:05.187682 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.185424    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7bab32b0-13c6-4055-916e-a7454bafb5b6-ovnkube-config\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.187682 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.185516    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7bab32b0-13c6-4055-916e-a7454bafb5b6-env-overrides\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.187682 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.185520    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bcf7b9b9-4c50-4c88-8190-4f62618f849d-etc-sysconfig\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.187682 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.185549    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bcf7b9b9-4c50-4c88-8190-4f62618f849d-sys\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.188617 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:05.185563    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f33e531-3341-4c01-940e-91f17519976c-metrics-certs podName:2f33e531-3341-4c01-940e-91f17519976c nodeName:}" failed. No retries permitted until 2026-07-28 22:05:05.685543609 +0000 UTC m=+3.065681238 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f33e531-3341-4c01-940e-91f17519976c-metrics-certs") pod "network-metrics-daemon-4xqhf" (UID: "2f33e531-3341-4c01-940e-91f17519976c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 22:05:05.188617 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.185597    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bcf7b9b9-4c50-4c88-8190-4f62618f849d-sys\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.188617 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.185598    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcf7b9b9-4c50-4c88-8190-4f62618f849d-var-lib-kubelet\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.188617 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.185639    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcf7b9b9-4c50-4c88-8190-4f62618f849d-var-lib-kubelet\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.188617 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.185657    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8347000f-940f-497d-af0c-71f180037642-host-slash\") pod \"iptables-alerter-pwnwp\" (UID: \"8347000f-940f-497d-af0c-71f180037642\") " pod="openshift-network-operator/iptables-alerter-pwnwp"
Jul 28 22:05:05.188617 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.185688    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-run-openvswitch\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.188617 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.185721    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bcf7b9b9-4c50-4c88-8190-4f62618f849d-etc-sysconfig\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.188617 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.185729    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-systemd-units\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.188617 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.185823    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7bab32b0-13c6-4055-916e-a7454bafb5b6-host-slash\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.188617 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.186019    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7bab32b0-13c6-4055-916e-a7454bafb5b6-ovnkube-script-lib\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.188617 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.186757    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7bab32b0-13c6-4055-916e-a7454bafb5b6-ovnkube-config\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.188617 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.187696    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6f95fc9f-b8c8-433e-ae9b-e20fcd167fbe-agent-certs\") pod \"konnectivity-agent-7wmm5\" (UID: \"6f95fc9f-b8c8-433e-ae9b-e20fcd167fbe\") " pod="kube-system/konnectivity-agent-7wmm5"
Jul 28 22:05:05.188617 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.187931    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcf7b9b9-4c50-4c88-8190-4f62618f849d-tmp\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.188617 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.188514    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bcf7b9b9-4c50-4c88-8190-4f62618f849d-etc-tuned\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.189318 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.188801    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7bab32b0-13c6-4055-916e-a7454bafb5b6-ovn-node-metrics-cert\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.198368 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.198328    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh5wd\" (UniqueName: \"kubernetes.io/projected/8347000f-940f-497d-af0c-71f180037642-kube-api-access-jh5wd\") pod \"iptables-alerter-pwnwp\" (UID: \"8347000f-940f-497d-af0c-71f180037642\") " pod="openshift-network-operator/iptables-alerter-pwnwp"
Jul 28 22:05:05.201346 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.201228    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zzcp\" (UniqueName: \"kubernetes.io/projected/7bab32b0-13c6-4055-916e-a7454bafb5b6-kube-api-access-7zzcp\") pod \"ovnkube-node-mcj8p\" (UID: \"7bab32b0-13c6-4055-916e-a7454bafb5b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.201475 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:05.201427    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 22:05:05.201475 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:05.201448    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 22:05:05.201475 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:05.201465    2563 projected.go:196] Error preparing data for projected volume kube-api-access-ntdgg for pod openshift-network-diagnostics/network-check-target-87zr2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 22:05:05.201659 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:05.201532    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/962b24a3-abd7-4a45-98fa-f225c383b0dc-kube-api-access-ntdgg podName:962b24a3-abd7-4a45-98fa-f225c383b0dc nodeName:}" failed. No retries permitted until 2026-07-28 22:05:05.701513524 +0000 UTC m=+3.081651153 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ntdgg" (UniqueName: "kubernetes.io/projected/962b24a3-abd7-4a45-98fa-f225c383b0dc-kube-api-access-ntdgg") pod "network-check-target-87zr2" (UID: "962b24a3-abd7-4a45-98fa-f225c383b0dc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 22:05:05.202929 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.202905    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwqhl\" (UniqueName: \"kubernetes.io/projected/bcf7b9b9-4c50-4c88-8190-4f62618f849d-kube-api-access-cwqhl\") pod \"tuned-jhwgt\" (UID: \"bcf7b9b9-4c50-4c88-8190-4f62618f849d\") " pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.203039 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.202986    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gvhx\" (UniqueName: \"kubernetes.io/projected/2f33e531-3341-4c01-940e-91f17519976c-kube-api-access-8gvhx\") pod \"network-metrics-daemon-4xqhf\" (UID: \"2f33e531-3341-4c01-940e-91f17519976c\") " pod="openshift-multus/network-metrics-daemon-4xqhf"
Jul 28 22:05:05.272042 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.272011    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pkhlk"
Jul 28 22:05:05.279220 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.279196    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qtgkr"
Jul 28 22:05:05.285622 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:05.285593    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bd94e5c_7c0a_47d0_8de0_9ac456d7f658.slice/crio-34bae63240fef8df7a00914e88190cb43daf10437b7bfbd6f209d5dce094b722 WatchSource:0}: Error finding container 34bae63240fef8df7a00914e88190cb43daf10437b7bfbd6f209d5dce094b722: Status 404 returned error can't find the container with id 34bae63240fef8df7a00914e88190cb43daf10437b7bfbd6f209d5dce094b722
Jul 28 22:05:05.287134 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.286943    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqvb4"
Jul 28 22:05:05.291331 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:05.290898    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod820ee419_dab7_43ef_94bf_cadbaf9a5232.slice/crio-cba91795e7445f1e2a5d0ade29a58587695314a81cb20d9a9a15e5cb4ea860ed WatchSource:0}: Error finding container cba91795e7445f1e2a5d0ade29a58587695314a81cb20d9a9a15e5cb4ea860ed: Status 404 returned error can't find the container with id cba91795e7445f1e2a5d0ade29a58587695314a81cb20d9a9a15e5cb4ea860ed
Jul 28 22:05:05.291331 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.290971    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tx5m6"
Jul 28 22:05:05.297918 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.297586    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4fjz6"
Jul 28 22:05:05.298156 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:05.298132    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaba445d_47e5_446e_9be4_9148e4d7e59a.slice/crio-cd8a8cd6d14fe306a9eebe0220987d485cba421a09615c5becc6f467129387b3 WatchSource:0}: Error finding container cd8a8cd6d14fe306a9eebe0220987d485cba421a09615c5becc6f467129387b3: Status 404 returned error can't find the container with id cd8a8cd6d14fe306a9eebe0220987d485cba421a09615c5becc6f467129387b3
Jul 28 22:05:05.302740 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:05.302710    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a9cac08_b862_4f7b_88e3_f4de98a035ea.slice/crio-f437a916545dfa80f282152536be2f9aa85b70ff1c75eeebe22e8a0a006be47a WatchSource:0}: Error finding container f437a916545dfa80f282152536be2f9aa85b70ff1c75eeebe22e8a0a006be47a: Status 404 returned error can't find the container with id f437a916545dfa80f282152536be2f9aa85b70ff1c75eeebe22e8a0a006be47a
Jul 28 22:05:05.304072 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.304040    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pwnwp"
Jul 28 22:05:05.308258 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:05.308233    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eea54c2_c579_4dab_8222_909d88ccf205.slice/crio-f30f257069d1b3629ae7fe86459b09b7709bca552bba06ef960f1c899adbbba6 WatchSource:0}: Error finding container f30f257069d1b3629ae7fe86459b09b7709bca552bba06ef960f1c899adbbba6: Status 404 returned error can't find the container with id f30f257069d1b3629ae7fe86459b09b7709bca552bba06ef960f1c899adbbba6
Jul 28 22:05:05.310153 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.310133    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:05.315804 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:05.315780    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8347000f_940f_497d_af0c_71f180037642.slice/crio-ff81b43c61a8f5392da32d23fd176d2f42fd565f01288045a24e5b87ab1308ed WatchSource:0}: Error finding container ff81b43c61a8f5392da32d23fd176d2f42fd565f01288045a24e5b87ab1308ed: Status 404 returned error can't find the container with id ff81b43c61a8f5392da32d23fd176d2f42fd565f01288045a24e5b87ab1308ed
Jul 28 22:05:05.315898 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.315863    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7wmm5"
Jul 28 22:05:05.321326 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.321304    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jhwgt"
Jul 28 22:05:05.322460 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:05.322415    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bab32b0_13c6_4055_916e_a7454bafb5b6.slice/crio-e0dd8cc168acefc1b7282b88aa9384c66fea39b554ef83bcafcd030e0aa54730 WatchSource:0}: Error finding container e0dd8cc168acefc1b7282b88aa9384c66fea39b554ef83bcafcd030e0aa54730: Status 404 returned error can't find the container with id e0dd8cc168acefc1b7282b88aa9384c66fea39b554ef83bcafcd030e0aa54730
Jul 28 22:05:05.325938 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:05.325919    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f95fc9f_b8c8_433e_ae9b_e20fcd167fbe.slice/crio-badb42c2eef1144b59c276828c2bc94537ff1f30f30c868d6095600cc90edaad WatchSource:0}: Error finding container badb42c2eef1144b59c276828c2bc94537ff1f30f30c868d6095600cc90edaad: Status 404 returned error can't find the container with id badb42c2eef1144b59c276828c2bc94537ff1f30f30c868d6095600cc90edaad
Jul 28 22:05:05.330158 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:05.330133    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcf7b9b9_4c50_4c88_8190_4f62618f849d.slice/crio-115f79eab2ad8576de620d8b1cf95873428c295f8ebaaddd668077716c29d226 WatchSource:0}: Error finding container 115f79eab2ad8576de620d8b1cf95873428c295f8ebaaddd668077716c29d226: Status 404 returned error can't find the container with id 115f79eab2ad8576de620d8b1cf95873428c295f8ebaaddd668077716c29d226
Jul 28 22:05:05.419126 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.419033    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 22:05:05.589417 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.589249    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d7dfdb2-a77a-47e8-865a-07a014ec5633-original-pull-secret\") pod \"global-pull-secret-syncer-2rqhg\" (UID: \"5d7dfdb2-a77a-47e8-865a-07a014ec5633\") " pod="kube-system/global-pull-secret-syncer-2rqhg"
Jul 28 22:05:05.589417 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:05.589409    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 22:05:05.589665 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:05.589500    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d7dfdb2-a77a-47e8-865a-07a014ec5633-original-pull-secret podName:5d7dfdb2-a77a-47e8-865a-07a014ec5633 nodeName:}" failed. No retries permitted until 2026-07-28 22:05:06.589475303 +0000 UTC m=+3.969612945 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d7dfdb2-a77a-47e8-865a-07a014ec5633-original-pull-secret") pod "global-pull-secret-syncer-2rqhg" (UID: "5d7dfdb2-a77a-47e8-865a-07a014ec5633") : object "kube-system"/"original-pull-secret" not registered
Jul 28 22:05:05.690664 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.690572    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f33e531-3341-4c01-940e-91f17519976c-metrics-certs\") pod \"network-metrics-daemon-4xqhf\" (UID: \"2f33e531-3341-4c01-940e-91f17519976c\") " pod="openshift-multus/network-metrics-daemon-4xqhf"
Jul 28 22:05:05.690821 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:05.690771    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 22:05:05.690880 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:05.690863    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f33e531-3341-4c01-940e-91f17519976c-metrics-certs podName:2f33e531-3341-4c01-940e-91f17519976c nodeName:}" failed. No retries permitted until 2026-07-28 22:05:06.690837719 +0000 UTC m=+4.070975343 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f33e531-3341-4c01-940e-91f17519976c-metrics-certs") pod "network-metrics-daemon-4xqhf" (UID: "2f33e531-3341-4c01-940e-91f17519976c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 22:05:05.791750 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:05.791705    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntdgg\" (UniqueName: \"kubernetes.io/projected/962b24a3-abd7-4a45-98fa-f225c383b0dc-kube-api-access-ntdgg\") pod \"network-check-target-87zr2\" (UID: \"962b24a3-abd7-4a45-98fa-f225c383b0dc\") " pod="openshift-network-diagnostics/network-check-target-87zr2"
Jul 28 22:05:05.791992 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:05.791909    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 22:05:05.791992 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:05.791932    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 22:05:05.791992 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:05.791947    2563 projected.go:196] Error preparing data for projected volume kube-api-access-ntdgg for pod openshift-network-diagnostics/network-check-target-87zr2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 22:05:05.792183 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:05.792018    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/962b24a3-abd7-4a45-98fa-f225c383b0dc-kube-api-access-ntdgg podName:962b24a3-abd7-4a45-98fa-f225c383b0dc nodeName:}" failed. No retries permitted until 2026-07-28 22:05:06.791998859 +0000 UTC m=+4.172136497 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ntdgg" (UniqueName: "kubernetes.io/projected/962b24a3-abd7-4a45-98fa-f225c383b0dc-kube-api-access-ntdgg") pod "network-check-target-87zr2" (UID: "962b24a3-abd7-4a45-98fa-f225c383b0dc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 22:05:06.015145 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:06.015073    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 22:05:06.023728 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:06.023680    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 22:00:05 +0000 UTC" deadline="2028-04-30 21:39:41.362748886 +0000 UTC"
Jul 28 22:05:06.023728 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:06.023725    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15407h34m35.339027304s"
Jul 28 22:05:06.124837 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:06.124799    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-87zr2"
Jul 28 22:05:06.125001 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:06.124967    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-87zr2" podUID="962b24a3-abd7-4a45-98fa-f225c383b0dc"
Jul 28 22:05:06.135254 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:06.135226    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jhwgt" event={"ID":"bcf7b9b9-4c50-4c88-8190-4f62618f849d","Type":"ContainerStarted","Data":"115f79eab2ad8576de620d8b1cf95873428c295f8ebaaddd668077716c29d226"}
Jul 28 22:05:06.136532 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:06.136501    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pwnwp" event={"ID":"8347000f-940f-497d-af0c-71f180037642","Type":"ContainerStarted","Data":"ff81b43c61a8f5392da32d23fd176d2f42fd565f01288045a24e5b87ab1308ed"}
Jul 28 22:05:06.137752 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:06.137728    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4fjz6" event={"ID":"6eea54c2-c579-4dab-8222-909d88ccf205","Type":"ContainerStarted","Data":"f30f257069d1b3629ae7fe86459b09b7709bca552bba06ef960f1c899adbbba6"}
Jul 28 22:05:06.138991 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:06.138958    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tx5m6" event={"ID":"2a9cac08-b862-4f7b-88e3-f4de98a035ea","Type":"ContainerStarted","Data":"f437a916545dfa80f282152536be2f9aa85b70ff1c75eeebe22e8a0a006be47a"}
Jul 28 22:05:06.140197 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:06.140168    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqvb4" event={"ID":"caba445d-47e5-446e-9be4-9148e4d7e59a","Type":"ContainerStarted","Data":"cd8a8cd6d14fe306a9eebe0220987d485cba421a09615c5becc6f467129387b3"}
Jul 28 22:05:06.141531 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:06.141508    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qtgkr" event={"ID":"820ee419-dab7-43ef-94bf-cadbaf9a5232","Type":"ContainerStarted","Data":"cba91795e7445f1e2a5d0ade29a58587695314a81cb20d9a9a15e5cb4ea860ed"}
Jul 28 22:05:06.142800 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:06.142770    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7wmm5" event={"ID":"6f95fc9f-b8c8-433e-ae9b-e20fcd167fbe","Type":"ContainerStarted","Data":"badb42c2eef1144b59c276828c2bc94537ff1f30f30c868d6095600cc90edaad"}
Jul 28 22:05:06.144081 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:06.144061    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p" event={"ID":"7bab32b0-13c6-4055-916e-a7454bafb5b6","Type":"ContainerStarted","Data":"e0dd8cc168acefc1b7282b88aa9384c66fea39b554ef83bcafcd030e0aa54730"}
Jul 28 22:05:06.145244 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:06.145207    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pkhlk" event={"ID":"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658","Type":"ContainerStarted","Data":"34bae63240fef8df7a00914e88190cb43daf10437b7bfbd6f209d5dce094b722"}
Jul 28 22:05:06.599998 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:06.599960    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d7dfdb2-a77a-47e8-865a-07a014ec5633-original-pull-secret\") pod \"global-pull-secret-syncer-2rqhg\" (UID: \"5d7dfdb2-a77a-47e8-865a-07a014ec5633\") " pod="kube-system/global-pull-secret-syncer-2rqhg"
Jul 28 22:05:06.600184 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:06.600155    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 22:05:06.600247 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:06.600219    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d7dfdb2-a77a-47e8-865a-07a014ec5633-original-pull-secret podName:5d7dfdb2-a77a-47e8-865a-07a014ec5633 nodeName:}" failed. No retries permitted until 2026-07-28 22:05:08.600199638 +0000 UTC m=+5.980337276 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d7dfdb2-a77a-47e8-865a-07a014ec5633-original-pull-secret") pod "global-pull-secret-syncer-2rqhg" (UID: "5d7dfdb2-a77a-47e8-865a-07a014ec5633") : object "kube-system"/"original-pull-secret" not registered
Jul 28 22:05:06.702079 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:06.701363    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f33e531-3341-4c01-940e-91f17519976c-metrics-certs\") pod \"network-metrics-daemon-4xqhf\" (UID: \"2f33e531-3341-4c01-940e-91f17519976c\") " pod="openshift-multus/network-metrics-daemon-4xqhf"
Jul 28 22:05:06.702079 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:06.701577    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 22:05:06.702079 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:06.701642    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f33e531-3341-4c01-940e-91f17519976c-metrics-certs podName:2f33e531-3341-4c01-940e-91f17519976c nodeName:}" failed. No retries permitted until 2026-07-28 22:05:08.701622195 +0000 UTC m=+6.081759841 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f33e531-3341-4c01-940e-91f17519976c-metrics-certs") pod "network-metrics-daemon-4xqhf" (UID: "2f33e531-3341-4c01-940e-91f17519976c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 22:05:06.801871 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:06.801806    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntdgg\" (UniqueName: \"kubernetes.io/projected/962b24a3-abd7-4a45-98fa-f225c383b0dc-kube-api-access-ntdgg\") pod \"network-check-target-87zr2\" (UID: \"962b24a3-abd7-4a45-98fa-f225c383b0dc\") " pod="openshift-network-diagnostics/network-check-target-87zr2"
Jul 28 22:05:06.802135 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:06.801971    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 22:05:06.802135 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:06.801992    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 22:05:06.802135 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:06.802005    2563 projected.go:196] Error preparing data for projected volume kube-api-access-ntdgg for pod openshift-network-diagnostics/network-check-target-87zr2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 22:05:06.802135 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:06.802062    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/962b24a3-abd7-4a45-98fa-f225c383b0dc-kube-api-access-ntdgg podName:962b24a3-abd7-4a45-98fa-f225c383b0dc nodeName:}" failed. No retries permitted until 2026-07-28 22:05:08.802042832 +0000 UTC m=+6.182180475 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ntdgg" (UniqueName: "kubernetes.io/projected/962b24a3-abd7-4a45-98fa-f225c383b0dc-kube-api-access-ntdgg") pod "network-check-target-87zr2" (UID: "962b24a3-abd7-4a45-98fa-f225c383b0dc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 22:05:07.023937 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:07.023840    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 22:00:05 +0000 UTC" deadline="2028-02-12 16:59:20.121668701 +0000 UTC"
Jul 28 22:05:07.023937 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:07.023933    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13530h54m13.097741221s"
Jul 28 22:05:07.124587 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:07.124551    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rqhg"
Jul 28 22:05:07.124767 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:07.124701    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rqhg" podUID="5d7dfdb2-a77a-47e8-865a-07a014ec5633"
Jul 28 22:05:07.125209 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:07.125185    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4xqhf"
Jul 28 22:05:07.125330 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:07.125308    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4xqhf" podUID="2f33e531-3341-4c01-940e-91f17519976c"
Jul 28 22:05:08.125280 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:08.124469    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-87zr2"
Jul 28 22:05:08.125280 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:08.124886    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-87zr2" podUID="962b24a3-abd7-4a45-98fa-f225c383b0dc"
Jul 28 22:05:08.156045 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:08.156009    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal" event={"ID":"e5ce7b7b59f5b560ebbfcc3f6f472e5b","Type":"ContainerStarted","Data":"1f037db3e89590533238d1960c3b694d76f8082f520dfb6c80a3ed005cb61ad8"}
Jul 28 22:05:08.167885 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:08.167246    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-37.ec2.internal" event={"ID":"2aaf76ae458a97a3e5edaa3e36196e52","Type":"ContainerStarted","Data":"da4553ebc931e2907e166700c7c5c2e8c3906b40a45aafe16949d03ad73617d5"}
Jul 28 22:05:08.620108 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:08.619466    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d7dfdb2-a77a-47e8-865a-07a014ec5633-original-pull-secret\") pod \"global-pull-secret-syncer-2rqhg\" (UID: \"5d7dfdb2-a77a-47e8-865a-07a014ec5633\") " pod="kube-system/global-pull-secret-syncer-2rqhg"
Jul 28 22:05:08.620108 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:08.619675    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 22:05:08.620108 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:08.619742    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d7dfdb2-a77a-47e8-865a-07a014ec5633-original-pull-secret podName:5d7dfdb2-a77a-47e8-865a-07a014ec5633 nodeName:}" failed. No retries permitted until 2026-07-28 22:05:12.61972232 +0000 UTC m=+9.999859960 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d7dfdb2-a77a-47e8-865a-07a014ec5633-original-pull-secret") pod "global-pull-secret-syncer-2rqhg" (UID: "5d7dfdb2-a77a-47e8-865a-07a014ec5633") : object "kube-system"/"original-pull-secret" not registered
Jul 28 22:05:08.720092 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:08.720054    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f33e531-3341-4c01-940e-91f17519976c-metrics-certs\") pod \"network-metrics-daemon-4xqhf\" (UID: \"2f33e531-3341-4c01-940e-91f17519976c\") " pod="openshift-multus/network-metrics-daemon-4xqhf"
Jul 28 22:05:08.720287 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:08.720216    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 22:05:08.720287 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:08.720281    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f33e531-3341-4c01-940e-91f17519976c-metrics-certs podName:2f33e531-3341-4c01-940e-91f17519976c nodeName:}" failed. No retries permitted until 2026-07-28 22:05:12.720259364 +0000 UTC m=+10.100396994 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f33e531-3341-4c01-940e-91f17519976c-metrics-certs") pod "network-metrics-daemon-4xqhf" (UID: "2f33e531-3341-4c01-940e-91f17519976c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 22:05:08.820852 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:08.820787    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntdgg\" (UniqueName: \"kubernetes.io/projected/962b24a3-abd7-4a45-98fa-f225c383b0dc-kube-api-access-ntdgg\") pod \"network-check-target-87zr2\" (UID: \"962b24a3-abd7-4a45-98fa-f225c383b0dc\") " pod="openshift-network-diagnostics/network-check-target-87zr2"
Jul 28 22:05:08.821055 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:08.820954    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 22:05:08.821055 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:08.820978    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 22:05:08.821055 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:08.820992    2563 projected.go:196] Error preparing data for projected volume kube-api-access-ntdgg for pod openshift-network-diagnostics/network-check-target-87zr2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 22:05:08.821055 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:08.821052    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/962b24a3-abd7-4a45-98fa-f225c383b0dc-kube-api-access-ntdgg podName:962b24a3-abd7-4a45-98fa-f225c383b0dc nodeName:}" failed. No retries permitted until 2026-07-28 22:05:12.821032513 +0000 UTC m=+10.201170139 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ntdgg" (UniqueName: "kubernetes.io/projected/962b24a3-abd7-4a45-98fa-f225c383b0dc-kube-api-access-ntdgg") pod "network-check-target-87zr2" (UID: "962b24a3-abd7-4a45-98fa-f225c383b0dc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 22:05:09.124550 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:09.124472    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rqhg"
Jul 28 22:05:09.124728 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:09.124622    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rqhg" podUID="5d7dfdb2-a77a-47e8-865a-07a014ec5633"
Jul 28 22:05:09.125170 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:09.124884    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4xqhf"
Jul 28 22:05:09.125170 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:09.125050    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4xqhf" podUID="2f33e531-3341-4c01-940e-91f17519976c"
Jul 28 22:05:09.173533 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:09.173474    2563 generic.go:356] "Generic (PLEG): container finished" podID="e5ce7b7b59f5b560ebbfcc3f6f472e5b" containerID="1f037db3e89590533238d1960c3b694d76f8082f520dfb6c80a3ed005cb61ad8" exitCode=0
Jul 28 22:05:09.174019 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:09.173599    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal" event={"ID":"e5ce7b7b59f5b560ebbfcc3f6f472e5b","Type":"ContainerDied","Data":"1f037db3e89590533238d1960c3b694d76f8082f520dfb6c80a3ed005cb61ad8"}
Jul 28 22:05:09.201643 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:09.201558    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-37.ec2.internal" podStartSLOduration=4.201534986 podStartE2EDuration="4.201534986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:05:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:05:08.193335761 +0000 UTC m=+5.573473412" watchObservedRunningTime="2026-07-28 22:05:09.201534986 +0000 UTC m=+6.581672637"
Jul 28 22:05:10.124886 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:10.124849    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-87zr2"
Jul 28 22:05:10.125093 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:10.124999    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-87zr2" podUID="962b24a3-abd7-4a45-98fa-f225c383b0dc"
Jul 28 22:05:11.125401 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:11.125343    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rqhg"
Jul 28 22:05:11.125886 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:11.125518    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rqhg" podUID="5d7dfdb2-a77a-47e8-865a-07a014ec5633"
Jul 28 22:05:11.125886 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:11.125568    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4xqhf"
Jul 28 22:05:11.125886 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:11.125755    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4xqhf" podUID="2f33e531-3341-4c01-940e-91f17519976c"
Jul 28 22:05:12.124767 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:12.124728    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-87zr2"
Jul 28 22:05:12.124968 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:12.124888    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-87zr2" podUID="962b24a3-abd7-4a45-98fa-f225c383b0dc"
Jul 28 22:05:12.659286 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:12.659243    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d7dfdb2-a77a-47e8-865a-07a014ec5633-original-pull-secret\") pod \"global-pull-secret-syncer-2rqhg\" (UID: \"5d7dfdb2-a77a-47e8-865a-07a014ec5633\") " pod="kube-system/global-pull-secret-syncer-2rqhg"
Jul 28 22:05:12.659783 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:12.659428    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 22:05:12.659783 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:12.659495    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d7dfdb2-a77a-47e8-865a-07a014ec5633-original-pull-secret podName:5d7dfdb2-a77a-47e8-865a-07a014ec5633 nodeName:}" failed. No retries permitted until 2026-07-28 22:05:20.659475703 +0000 UTC m=+18.039613330 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d7dfdb2-a77a-47e8-865a-07a014ec5633-original-pull-secret") pod "global-pull-secret-syncer-2rqhg" (UID: "5d7dfdb2-a77a-47e8-865a-07a014ec5633") : object "kube-system"/"original-pull-secret" not registered
Jul 28 22:05:12.760815 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:12.760769    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f33e531-3341-4c01-940e-91f17519976c-metrics-certs\") pod \"network-metrics-daemon-4xqhf\" (UID: \"2f33e531-3341-4c01-940e-91f17519976c\") " pod="openshift-multus/network-metrics-daemon-4xqhf"
Jul 28 22:05:12.761007 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:12.760968    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 22:05:12.761085 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:12.761056    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f33e531-3341-4c01-940e-91f17519976c-metrics-certs podName:2f33e531-3341-4c01-940e-91f17519976c nodeName:}" failed. No retries permitted until 2026-07-28 22:05:20.761033337 +0000 UTC m=+18.141170967 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f33e531-3341-4c01-940e-91f17519976c-metrics-certs") pod "network-metrics-daemon-4xqhf" (UID: "2f33e531-3341-4c01-940e-91f17519976c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 22:05:12.862193 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:12.861524    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntdgg\" (UniqueName: \"kubernetes.io/projected/962b24a3-abd7-4a45-98fa-f225c383b0dc-kube-api-access-ntdgg\") pod \"network-check-target-87zr2\" (UID: \"962b24a3-abd7-4a45-98fa-f225c383b0dc\") " pod="openshift-network-diagnostics/network-check-target-87zr2"
Jul 28 22:05:12.862193 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:12.861711    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 22:05:12.862193 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:12.861731    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 22:05:12.862193 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:12.861746    2563 projected.go:196] Error preparing data for projected volume kube-api-access-ntdgg for pod openshift-network-diagnostics/network-check-target-87zr2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 22:05:12.862193 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:12.861815    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/962b24a3-abd7-4a45-98fa-f225c383b0dc-kube-api-access-ntdgg podName:962b24a3-abd7-4a45-98fa-f225c383b0dc nodeName:}" failed. No retries permitted until 2026-07-28 22:05:20.861795894 +0000 UTC m=+18.241933534 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ntdgg" (UniqueName: "kubernetes.io/projected/962b24a3-abd7-4a45-98fa-f225c383b0dc-kube-api-access-ntdgg") pod "network-check-target-87zr2" (UID: "962b24a3-abd7-4a45-98fa-f225c383b0dc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 22:05:13.126520 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:13.125888    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rqhg"
Jul 28 22:05:13.126520 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:13.126005    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rqhg" podUID="5d7dfdb2-a77a-47e8-865a-07a014ec5633"
Jul 28 22:05:13.126520 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:13.126481    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4xqhf"
Jul 28 22:05:13.126810 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:13.126616    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4xqhf" podUID="2f33e531-3341-4c01-940e-91f17519976c"
Jul 28 22:05:14.124793 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:14.124699    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-87zr2"
Jul 28 22:05:14.125267 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:14.124860    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-87zr2" podUID="962b24a3-abd7-4a45-98fa-f225c383b0dc"
Jul 28 22:05:15.124401 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:15.124299    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4xqhf"
Jul 28 22:05:15.124401 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:15.124345    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rqhg"
Jul 28 22:05:15.124707 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:15.124484    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4xqhf" podUID="2f33e531-3341-4c01-940e-91f17519976c"
Jul 28 22:05:15.124707 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:15.124582    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rqhg" podUID="5d7dfdb2-a77a-47e8-865a-07a014ec5633"
Jul 28 22:05:16.124166 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:16.124135    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-87zr2"
Jul 28 22:05:16.124634 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:16.124249    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-87zr2" podUID="962b24a3-abd7-4a45-98fa-f225c383b0dc"
Jul 28 22:05:17.124893 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:17.124818    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rqhg"
Jul 28 22:05:17.124893 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:17.124846    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4xqhf"
Jul 28 22:05:17.125398 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:17.124957    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rqhg" podUID="5d7dfdb2-a77a-47e8-865a-07a014ec5633"
Jul 28 22:05:17.125398 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:17.125128    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4xqhf" podUID="2f33e531-3341-4c01-940e-91f17519976c"
Jul 28 22:05:18.124853 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:18.124810    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-87zr2"
Jul 28 22:05:18.125045 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:18.124964    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-87zr2" podUID="962b24a3-abd7-4a45-98fa-f225c383b0dc"
Jul 28 22:05:19.124907 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:19.124874    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4xqhf"
Jul 28 22:05:19.125077 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:19.124922    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rqhg"
Jul 28 22:05:19.125077 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:19.125014    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4xqhf" podUID="2f33e531-3341-4c01-940e-91f17519976c"
Jul 28 22:05:19.125485 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:19.125134    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rqhg" podUID="5d7dfdb2-a77a-47e8-865a-07a014ec5633"
Jul 28 22:05:20.124779 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:20.124740    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-87zr2"
Jul 28 22:05:20.124955 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:20.124884    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-87zr2" podUID="962b24a3-abd7-4a45-98fa-f225c383b0dc"
Jul 28 22:05:20.727548 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:20.727515    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d7dfdb2-a77a-47e8-865a-07a014ec5633-original-pull-secret\") pod \"global-pull-secret-syncer-2rqhg\" (UID: \"5d7dfdb2-a77a-47e8-865a-07a014ec5633\") " pod="kube-system/global-pull-secret-syncer-2rqhg"
Jul 28 22:05:20.727946 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:20.727669    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 22:05:20.727946 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:20.727737    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d7dfdb2-a77a-47e8-865a-07a014ec5633-original-pull-secret podName:5d7dfdb2-a77a-47e8-865a-07a014ec5633 nodeName:}" failed. No retries permitted until 2026-07-28 22:05:36.727722551 +0000 UTC m=+34.107860177 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d7dfdb2-a77a-47e8-865a-07a014ec5633-original-pull-secret") pod "global-pull-secret-syncer-2rqhg" (UID: "5d7dfdb2-a77a-47e8-865a-07a014ec5633") : object "kube-system"/"original-pull-secret" not registered
Jul 28 22:05:20.827929 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:20.827898    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f33e531-3341-4c01-940e-91f17519976c-metrics-certs\") pod \"network-metrics-daemon-4xqhf\" (UID: \"2f33e531-3341-4c01-940e-91f17519976c\") " pod="openshift-multus/network-metrics-daemon-4xqhf"
Jul 28 22:05:20.828083 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:20.828064    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 22:05:20.828146 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:20.828138    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f33e531-3341-4c01-940e-91f17519976c-metrics-certs podName:2f33e531-3341-4c01-940e-91f17519976c nodeName:}" failed. No retries permitted until 2026-07-28 22:05:36.828116667 +0000 UTC m=+34.208254292 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f33e531-3341-4c01-940e-91f17519976c-metrics-certs") pod "network-metrics-daemon-4xqhf" (UID: "2f33e531-3341-4c01-940e-91f17519976c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 22:05:20.928338 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:20.928308    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntdgg\" (UniqueName: \"kubernetes.io/projected/962b24a3-abd7-4a45-98fa-f225c383b0dc-kube-api-access-ntdgg\") pod \"network-check-target-87zr2\" (UID: \"962b24a3-abd7-4a45-98fa-f225c383b0dc\") " pod="openshift-network-diagnostics/network-check-target-87zr2"
Jul 28 22:05:20.928492 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:20.928448    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 22:05:20.928492 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:20.928462    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 22:05:20.928492 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:20.928471    2563 projected.go:196] Error preparing data for projected volume kube-api-access-ntdgg for pod openshift-network-diagnostics/network-check-target-87zr2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 22:05:20.928602 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:20.928520    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/962b24a3-abd7-4a45-98fa-f225c383b0dc-kube-api-access-ntdgg podName:962b24a3-abd7-4a45-98fa-f225c383b0dc nodeName:}" failed. No retries permitted until 2026-07-28 22:05:36.928508031 +0000 UTC m=+34.308645653 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ntdgg" (UniqueName: "kubernetes.io/projected/962b24a3-abd7-4a45-98fa-f225c383b0dc-kube-api-access-ntdgg") pod "network-check-target-87zr2" (UID: "962b24a3-abd7-4a45-98fa-f225c383b0dc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 22:05:21.124239 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:21.124129    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rqhg"
Jul 28 22:05:21.124418 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:21.124273    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rqhg" podUID="5d7dfdb2-a77a-47e8-865a-07a014ec5633"
Jul 28 22:05:21.124418 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:21.124321    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4xqhf"
Jul 28 22:05:21.124583 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:21.124466    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4xqhf" podUID="2f33e531-3341-4c01-940e-91f17519976c"
Jul 28 22:05:22.124534 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:22.124498    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-87zr2"
Jul 28 22:05:22.125059 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:22.124645    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-87zr2" podUID="962b24a3-abd7-4a45-98fa-f225c383b0dc"
Jul 28 22:05:23.125299 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:23.125260    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rqhg"
Jul 28 22:05:23.125760 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:23.125384    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rqhg" podUID="5d7dfdb2-a77a-47e8-865a-07a014ec5633"
Jul 28 22:05:23.125760 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:23.125422    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4xqhf"
Jul 28 22:05:23.125760 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:23.125525    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4xqhf" podUID="2f33e531-3341-4c01-940e-91f17519976c"
Jul 28 22:05:24.124102 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:24.124059    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-87zr2"
Jul 28 22:05:24.124300 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:24.124186    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-87zr2" podUID="962b24a3-abd7-4a45-98fa-f225c383b0dc"
Jul 28 22:05:25.124173 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:25.124151    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rqhg"
Jul 28 22:05:25.124539 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:25.124284    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rqhg" podUID="5d7dfdb2-a77a-47e8-865a-07a014ec5633"
Jul 28 22:05:25.124539 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:25.124423    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4xqhf"
Jul 28 22:05:25.124539 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:25.124519    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4xqhf" podUID="2f33e531-3341-4c01-940e-91f17519976c"
Jul 28 22:05:25.205794 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:25.205690    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pkhlk" event={"ID":"1bd94e5c-7c0a-47d0-8de0-9ac456d7f658","Type":"ContainerStarted","Data":"19215a5757f6b692c59efd71ca4a1f7d1c03c1353d54ce0135ec1a05350556eb"}
Jul 28 22:05:25.212848 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:25.212513    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal" event={"ID":"e5ce7b7b59f5b560ebbfcc3f6f472e5b","Type":"ContainerStarted","Data":"273a8dd0787152302c5338e03a2e22d1e623de5aa8ac012dbca24ded620a42dd"}
Jul 28 22:05:25.215005 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:25.214976    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jhwgt" event={"ID":"bcf7b9b9-4c50-4c88-8190-4f62618f849d","Type":"ContainerStarted","Data":"0c79b0f7a69417f7923342c101f63942d891bd33cffd1bb5514d5d29f17e9b8f"}
Jul 28 22:05:25.217435 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:25.217343    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4fjz6" event={"ID":"6eea54c2-c579-4dab-8222-909d88ccf205","Type":"ContainerStarted","Data":"3cbcef8c3ce854f04a40ef43a1eca18f5147fcc3b0afe5e46736eb2b2ff9fcbb"}
Jul 28 22:05:25.219778 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:25.219748    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tx5m6" event={"ID":"2a9cac08-b862-4f7b-88e3-f4de98a035ea","Type":"ContainerStarted","Data":"b68d03f8be31c04ffc9d0b1cf72353a2ac1e6d77e490dc8c1aacb5b688e727c7"}
Jul 28 22:05:25.221557 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:25.221514    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqvb4" event={"ID":"caba445d-47e5-446e-9be4-9148e4d7e59a","Type":"ContainerStarted","Data":"58ea09542eaf2dead944221aab2ac60be05aec878b19536145d01ded22ea06f9"}
Jul 28 22:05:25.223101 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:25.223070    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qtgkr" event={"ID":"820ee419-dab7-43ef-94bf-cadbaf9a5232","Type":"ContainerStarted","Data":"862b7765098c3aa7eb71c54897241b4bcbaeb1dc954426f22d9c7f617675d831"}
Jul 28 22:05:25.224613 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:25.224589    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7wmm5" event={"ID":"6f95fc9f-b8c8-433e-ae9b-e20fcd167fbe","Type":"ContainerStarted","Data":"1c825e12823a7d063512b2fbba02eff2fa49edd125b6fd0e491337fb60822178"}
Jul 28 22:05:25.280912 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:25.280869    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-pkhlk" podStartSLOduration=1.593132936 podStartE2EDuration="21.280852642s" totalImagesPullingTime="19.687719706s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:05:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:05:25.254310484 +0000 UTC m=+22.634448128" watchObservedRunningTime="2026-07-28 22:05:25.280852642 +0000 UTC m=+22.660990287"
Jul 28 22:05:25.281112 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:25.281092    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jhwgt" podStartSLOduration=1.6767133950000002 podStartE2EDuration="21.281086001s" totalImagesPullingTime="19.604372606s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:05:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:05:25.28099282 +0000 UTC m=+22.661130452" watchObservedRunningTime="2026-07-28 22:05:25.281086001 +0000 UTC m=+22.661223644"
Jul 28 22:05:25.323811 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:25.323766    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-37.ec2.internal" podStartSLOduration=20.323749064 podStartE2EDuration="20.323749064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:05:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:05:25.303914634 +0000 UTC m=+22.684052277" watchObservedRunningTime="2026-07-28 22:05:25.323749064 +0000 UTC m=+22.703886730"
Jul 28 22:05:25.347181 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:25.347139    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qtgkr" podStartSLOduration=1.707404419 podStartE2EDuration="21.34711986s" totalImagesPullingTime="19.639715441s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:05:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:05:25.324122482 +0000 UTC m=+22.704260125" watchObservedRunningTime="2026-07-28 22:05:25.34711986 +0000 UTC m=+22.727257503"
Jul 28 22:05:25.347583 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:25.347557    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tx5m6" podStartSLOduration=1.7187552240000001 podStartE2EDuration="21.347546718s" totalImagesPullingTime="19.628791494s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:05:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:05:25.347395682 +0000 UTC m=+22.727533327" watchObservedRunningTime="2026-07-28 22:05:25.347546718 +0000 UTC m=+22.727684362"
Jul 28 22:05:25.415259 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:25.415215    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7wmm5" podStartSLOduration=5.99397492 podStartE2EDuration="21.41519994s" totalImagesPullingTime="15.42122502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:05:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:05:25.414120349 +0000 UTC m=+22.794257993" watchObservedRunningTime="2026-07-28 22:05:25.41519994 +0000 UTC m=+22.795337583"
Jul 28 22:05:26.124482 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:26.124280    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-87zr2"
Jul 28 22:05:26.125246 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:26.124577    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-87zr2" podUID="962b24a3-abd7-4a45-98fa-f225c383b0dc"
Jul 28 22:05:26.227792 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:26.227766    2563 generic.go:356] "Generic (PLEG): container finished" podID="6eea54c2-c579-4dab-8222-909d88ccf205" containerID="3cbcef8c3ce854f04a40ef43a1eca18f5147fcc3b0afe5e46736eb2b2ff9fcbb" exitCode=0
Jul 28 22:05:26.227916 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:26.227848    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4fjz6" event={"ID":"6eea54c2-c579-4dab-8222-909d88ccf205","Type":"ContainerDied","Data":"3cbcef8c3ce854f04a40ef43a1eca18f5147fcc3b0afe5e46736eb2b2ff9fcbb"}
Jul 28 22:05:26.230588 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:26.230545    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mcj8p_7bab32b0-13c6-4055-916e-a7454bafb5b6/ovn-acl-logging/0.log"
Jul 28 22:05:26.230887 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:26.230861    2563 generic.go:356] "Generic (PLEG): container finished" podID="7bab32b0-13c6-4055-916e-a7454bafb5b6" containerID="e57aab91edb2edc0df56d72472a5035e8f2d28d0cde4372784f98ca59fd15491" exitCode=1
Jul 28 22:05:26.230965 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:26.230949    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p" event={"ID":"7bab32b0-13c6-4055-916e-a7454bafb5b6","Type":"ContainerStarted","Data":"afd8062c7e3a25fd94669da7e550a736cd82bc961e715167793aa03d7ea96b2d"}
Jul 28 22:05:26.231049 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:26.231033    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p" event={"ID":"7bab32b0-13c6-4055-916e-a7454bafb5b6","Type":"ContainerStarted","Data":"8e3229414e986010a9994dd9b103679583fa74b542bda7f310c05df795411a39"}
Jul 28 22:05:26.231104 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:26.231090    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p" event={"ID":"7bab32b0-13c6-4055-916e-a7454bafb5b6","Type":"ContainerStarted","Data":"558d626a55debd05f9abbbbc68d7f03efa99c2f8d1b3cd3a121d6d2e800442d1"}
Jul 28 22:05:26.231159 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:26.231145    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p" event={"ID":"7bab32b0-13c6-4055-916e-a7454bafb5b6","Type":"ContainerStarted","Data":"59c5e659cbde8c1b0d28ca3ba927222d7398585a7329a5526c0eb60d0a358e72"}
Jul 28 22:05:26.231222 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:26.231209    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p" event={"ID":"7bab32b0-13c6-4055-916e-a7454bafb5b6","Type":"ContainerDied","Data":"e57aab91edb2edc0df56d72472a5035e8f2d28d0cde4372784f98ca59fd15491"}
Jul 28 22:05:26.231280 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:26.231267    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p" event={"ID":"7bab32b0-13c6-4055-916e-a7454bafb5b6","Type":"ContainerStarted","Data":"083b3d366970324a96bf4170a70bcbe6a9aebdfca2517af47664e929e033a8f2"}
Jul 28 22:05:26.300265 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:26.300243    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 22:05:27.035768 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:27.035633    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T22:05:26.300260321Z","UUID":"7fff32bb-1048-4a39-837e-73101c7356ce","Handler":null,"Name":"","Endpoint":""}
Jul 28 22:05:27.037521 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:27.037494    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 22:05:27.037657 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:27.037537    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 22:05:27.124151 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:27.124113    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4xqhf"
Jul 28 22:05:27.124324 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:27.124113    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rqhg"
Jul 28 22:05:27.124324 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:27.124234    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4xqhf" podUID="2f33e531-3341-4c01-940e-91f17519976c"
Jul 28 22:05:27.124458 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:27.124373    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rqhg" podUID="5d7dfdb2-a77a-47e8-865a-07a014ec5633"
Jul 28 22:05:27.235310 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:27.235271    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pwnwp" event={"ID":"8347000f-940f-497d-af0c-71f180037642","Type":"ContainerStarted","Data":"526334442c660fa21c652ed93760b352d9f488f67d39314375813fe1e02886fa"}
Jul 28 22:05:27.237345 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:27.237315    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqvb4" event={"ID":"caba445d-47e5-446e-9be4-9148e4d7e59a","Type":"ContainerStarted","Data":"c72d61db3c67fd90f054ec5886fce803651832a4ef99d35be9b69696fb727fd1"}
Jul 28 22:05:27.254720 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:27.254696    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7wmm5"
Jul 28 22:05:27.255644 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:27.255532    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7wmm5"
Jul 28 22:05:27.275689 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:27.275640    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pwnwp" podStartSLOduration=3.6604474959999997 podStartE2EDuration="23.27562113s" totalImagesPullingTime="19.615173634s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:05:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:05:27.257877362 +0000 UTC m=+24.638015009" watchObservedRunningTime="2026-07-28 22:05:27.27562113 +0000 UTC m=+24.655758775"
Jul 28 22:05:28.124298 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:28.124026    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-87zr2"
Jul 28 22:05:28.124463 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:28.124370    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-87zr2" podUID="962b24a3-abd7-4a45-98fa-f225c383b0dc"
Jul 28 22:05:28.242574 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:28.242531    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqvb4" event={"ID":"caba445d-47e5-446e-9be4-9148e4d7e59a","Type":"ContainerStarted","Data":"f845719b7ea3264bf02200321d894e3dfc1a9bcd498653355d99237f2b155887"}
Jul 28 22:05:28.245956 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:28.245934    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mcj8p_7bab32b0-13c6-4055-916e-a7454bafb5b6/ovn-acl-logging/0.log"
Jul 28 22:05:28.246376 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:28.246336    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p" event={"ID":"7bab32b0-13c6-4055-916e-a7454bafb5b6","Type":"ContainerStarted","Data":"c6878af1c994efa6cb759d167a1a95da8143f09bf4bbf6629e8b448e08c0a750"}
Jul 28 22:05:28.246636 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:28.246615    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7wmm5"
Jul 28 22:05:28.247070 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:28.247054    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7wmm5"
Jul 28 22:05:28.262095 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:28.262046    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqvb4" podStartSLOduration=2.68741882 podStartE2EDuration="24.262031597s" totalImagesPullingTime="21.574612777s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:05:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:05:28.261412655 +0000 UTC m=+25.641550303" watchObservedRunningTime="2026-07-28 22:05:28.262031597 +0000 UTC m=+25.642169242"
Jul 28 22:05:29.124829 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:29.124797    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rqhg"
Jul 28 22:05:29.125038 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:29.124917    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rqhg" podUID="5d7dfdb2-a77a-47e8-865a-07a014ec5633"
Jul 28 22:05:29.125038 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:29.124967    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4xqhf"
Jul 28 22:05:29.125142 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:29.125088    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4xqhf" podUID="2f33e531-3341-4c01-940e-91f17519976c"
Jul 28 22:05:30.124537 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:30.124514    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-87zr2"
Jul 28 22:05:30.125218 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:30.124637    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-87zr2" podUID="962b24a3-abd7-4a45-98fa-f225c383b0dc"
Jul 28 22:05:30.254307 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:30.254149    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mcj8p_7bab32b0-13c6-4055-916e-a7454bafb5b6/ovn-acl-logging/0.log"
Jul 28 22:05:30.254694 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:30.254668    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p" event={"ID":"7bab32b0-13c6-4055-916e-a7454bafb5b6","Type":"ContainerStarted","Data":"6921ec2cf79c49a2d1777fb8f80961149081accdb39e7d31dc5703cd852f4c4f"}
Jul 28 22:05:30.258383 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:30.255936    2563 scope.go:117] "RemoveContainer" containerID="e57aab91edb2edc0df56d72472a5035e8f2d28d0cde4372784f98ca59fd15491"
Jul 28 22:05:31.124579 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:31.124552    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rqhg"
Jul 28 22:05:31.125050 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:31.124657    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rqhg" podUID="5d7dfdb2-a77a-47e8-865a-07a014ec5633"
Jul 28 22:05:31.125050 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:31.124761    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4xqhf"
Jul 28 22:05:31.125050 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:31.124892    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4xqhf" podUID="2f33e531-3341-4c01-940e-91f17519976c"
Jul 28 22:05:31.260515 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:31.260493    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mcj8p_7bab32b0-13c6-4055-916e-a7454bafb5b6/ovn-acl-logging/0.log"
Jul 28 22:05:31.260857 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:31.260835    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p" event={"ID":"7bab32b0-13c6-4055-916e-a7454bafb5b6","Type":"ContainerStarted","Data":"7f980bfbc5b85650c6ebfd349652f3e6871f8c074e034f9d2db64139f5b54f6a"}
Jul 28 22:05:31.261130 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:31.261108    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:31.261245 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:31.261180    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:31.261245 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:31.261232    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:31.262647 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:31.262626    2563 generic.go:356] "Generic (PLEG): container finished" podID="6eea54c2-c579-4dab-8222-909d88ccf205" containerID="2016c9c96d523815d6895b259436cfb77e1854cbfd443f2796a67440c130a6f8" exitCode=0
Jul 28 22:05:31.262734 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:31.262656    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4fjz6" event={"ID":"6eea54c2-c579-4dab-8222-909d88ccf205","Type":"ContainerDied","Data":"2016c9c96d523815d6895b259436cfb77e1854cbfd443f2796a67440c130a6f8"}
Jul 28 22:05:31.276748 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:31.276725    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:31.276937 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:31.276921    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:05:31.288518 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:31.288482    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p" podStartSLOduration=7.4128601100000004 podStartE2EDuration="27.28846929s" totalImagesPullingTime="19.87560918s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:05:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:05:31.286865644 +0000 UTC m=+28.667003288" watchObservedRunningTime="2026-07-28 22:05:31.28846929 +0000 UTC m=+28.668606933"
Jul 28 22:05:32.124989 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:32.124958    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-87zr2"
Jul 28 22:05:32.125433 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:32.125079    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-87zr2" podUID="962b24a3-abd7-4a45-98fa-f225c383b0dc"
Jul 28 22:05:32.359741 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:32.359579    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-87zr2"]
Jul 28 22:05:32.359845 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:32.359832    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-87zr2"
Jul 28 22:05:32.359957 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:32.359937    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-87zr2" podUID="962b24a3-abd7-4a45-98fa-f225c383b0dc"
Jul 28 22:05:32.362556 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:32.362531    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4xqhf"]
Jul 28 22:05:32.362669 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:32.362656    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4xqhf"
Jul 28 22:05:32.362796 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:32.362773    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4xqhf" podUID="2f33e531-3341-4c01-940e-91f17519976c"
Jul 28 22:05:32.378326 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:32.378307    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2rqhg"]
Jul 28 22:05:32.378433 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:32.378420    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rqhg"
Jul 28 22:05:32.378545 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:32.378522    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rqhg" podUID="5d7dfdb2-a77a-47e8-865a-07a014ec5633"
Jul 28 22:05:33.269033 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:33.269002    2563 generic.go:356] "Generic (PLEG): container finished" podID="6eea54c2-c579-4dab-8222-909d88ccf205" containerID="7985cbb4d6a577cdde5fb37f85064ec1675b0332ab0f5b35164ae1fa8297370e" exitCode=0
Jul 28 22:05:33.269650 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:33.269081    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4fjz6" event={"ID":"6eea54c2-c579-4dab-8222-909d88ccf205","Type":"ContainerDied","Data":"7985cbb4d6a577cdde5fb37f85064ec1675b0332ab0f5b35164ae1fa8297370e"}
Jul 28 22:05:34.124208 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:34.124174    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-87zr2"
Jul 28 22:05:34.124208 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:34.124206    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rqhg"
Jul 28 22:05:34.124413 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:34.124208    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4xqhf"
Jul 28 22:05:34.124413 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:34.124310    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-87zr2" podUID="962b24a3-abd7-4a45-98fa-f225c383b0dc"
Jul 28 22:05:34.124532 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:34.124407    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rqhg" podUID="5d7dfdb2-a77a-47e8-865a-07a014ec5633"
Jul 28 22:05:34.124532 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:34.124507    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4xqhf" podUID="2f33e531-3341-4c01-940e-91f17519976c"
Jul 28 22:05:35.275130 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:35.275045    2563 generic.go:356] "Generic (PLEG): container finished" podID="6eea54c2-c579-4dab-8222-909d88ccf205" containerID="38c142d57bb5abb4021c828aeb7b30e45b5a54862444e5d1f1ee2b7f8b7943e0" exitCode=0
Jul 28 22:05:35.275130 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:35.275120    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4fjz6" event={"ID":"6eea54c2-c579-4dab-8222-909d88ccf205","Type":"ContainerDied","Data":"38c142d57bb5abb4021c828aeb7b30e45b5a54862444e5d1f1ee2b7f8b7943e0"}
Jul 28 22:05:36.124451 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:36.124416    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-87zr2"
Jul 28 22:05:36.124606 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:36.124425    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4xqhf"
Jul 28 22:05:36.124606 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:36.124554    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-87zr2" podUID="962b24a3-abd7-4a45-98fa-f225c383b0dc"
Jul 28 22:05:36.124606 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:36.124433    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rqhg"
Jul 28 22:05:36.124762 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:36.124595    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4xqhf" podUID="2f33e531-3341-4c01-940e-91f17519976c"
Jul 28 22:05:36.124762 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:36.124716    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rqhg" podUID="5d7dfdb2-a77a-47e8-865a-07a014ec5633"
Jul 28 22:05:36.748329 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:36.748295    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d7dfdb2-a77a-47e8-865a-07a014ec5633-original-pull-secret\") pod \"global-pull-secret-syncer-2rqhg\" (UID: \"5d7dfdb2-a77a-47e8-865a-07a014ec5633\") " pod="kube-system/global-pull-secret-syncer-2rqhg"
Jul 28 22:05:36.748764 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:36.748453    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 22:05:36.748764 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:36.748533    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d7dfdb2-a77a-47e8-865a-07a014ec5633-original-pull-secret podName:5d7dfdb2-a77a-47e8-865a-07a014ec5633 nodeName:}" failed. No retries permitted until 2026-07-28 22:06:08.74851277 +0000 UTC m=+66.128650398 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d7dfdb2-a77a-47e8-865a-07a014ec5633-original-pull-secret") pod "global-pull-secret-syncer-2rqhg" (UID: "5d7dfdb2-a77a-47e8-865a-07a014ec5633") : object "kube-system"/"original-pull-secret" not registered
Jul 28 22:05:36.849175 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:36.849130    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f33e531-3341-4c01-940e-91f17519976c-metrics-certs\") pod \"network-metrics-daemon-4xqhf\" (UID: \"2f33e531-3341-4c01-940e-91f17519976c\") " pod="openshift-multus/network-metrics-daemon-4xqhf"
Jul 28 22:05:36.849366 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:36.849306    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 22:05:36.849435 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:36.849404    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f33e531-3341-4c01-940e-91f17519976c-metrics-certs podName:2f33e531-3341-4c01-940e-91f17519976c nodeName:}" failed. No retries permitted until 2026-07-28 22:06:08.849383208 +0000 UTC m=+66.229520845 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f33e531-3341-4c01-940e-91f17519976c-metrics-certs") pod "network-metrics-daemon-4xqhf" (UID: "2f33e531-3341-4c01-940e-91f17519976c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 22:05:36.950315 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:36.950272    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntdgg\" (UniqueName: \"kubernetes.io/projected/962b24a3-abd7-4a45-98fa-f225c383b0dc-kube-api-access-ntdgg\") pod \"network-check-target-87zr2\" (UID: \"962b24a3-abd7-4a45-98fa-f225c383b0dc\") " pod="openshift-network-diagnostics/network-check-target-87zr2"
Jul 28 22:05:36.950466 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:36.950414    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 22:05:36.950466 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:36.950432    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 22:05:36.950466 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:36.950441    2563 projected.go:196] Error preparing data for projected volume kube-api-access-ntdgg for pod openshift-network-diagnostics/network-check-target-87zr2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 22:05:36.950583 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:36.950493    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/962b24a3-abd7-4a45-98fa-f225c383b0dc-kube-api-access-ntdgg podName:962b24a3-abd7-4a45-98fa-f225c383b0dc nodeName:}" failed. No retries permitted until 2026-07-28 22:06:08.950475222 +0000 UTC m=+66.330612844 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ntdgg" (UniqueName: "kubernetes.io/projected/962b24a3-abd7-4a45-98fa-f225c383b0dc-kube-api-access-ntdgg") pod "network-check-target-87zr2" (UID: "962b24a3-abd7-4a45-98fa-f225c383b0dc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 22:05:37.999723 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:37.999694    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-37.ec2.internal" event="NodeReady"
Jul 28 22:05:38.000173 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:37.999834    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 22:05:38.045701 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.045504    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vqx8j"]
Jul 28 22:05:38.048179 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.048157    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vqx8j"
Jul 28 22:05:38.051824 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.051549    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 22:05:38.051824 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.051552    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 22:05:38.052077 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.052048    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rvh82\""
Jul 28 22:05:38.052452 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.052422    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ssgsl"]
Jul 28 22:05:38.054245 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.054227    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ssgsl"
Jul 28 22:05:38.056889 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.056772    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 22:05:38.056960 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.056950    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 22:05:38.057184 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.057163    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 22:05:38.057282 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.057217    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-c8bjt\""
Jul 28 22:05:38.058795 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.058766    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vqx8j"]
Jul 28 22:05:38.067257 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.067238    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ssgsl"]
Jul 28 22:05:38.123963 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.123923    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4xqhf"
Jul 28 22:05:38.123963 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.123961    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rqhg"
Jul 28 22:05:38.124199 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.123960    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-87zr2"
Jul 28 22:05:38.126909 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.126884    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 22:05:38.127031 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.126930    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f5pt7\""
Jul 28 22:05:38.127031 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.126981    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 22:05:38.127140 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.127042    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-65j7n\""
Jul 28 22:05:38.127140 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.127125    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 22:05:38.127762 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.127746    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 22:05:38.158527 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.158503    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/331ef64d-2011-4149-b38c-88921f08f078-config-volume\") pod \"dns-default-vqx8j\" (UID: \"331ef64d-2011-4149-b38c-88921f08f078\") " pod="openshift-dns/dns-default-vqx8j"
Jul 28 22:05:38.158641 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.158550    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68d64\" (UniqueName: \"kubernetes.io/projected/d653889b-b9a3-4d5d-bbf0-f286413a20c9-kube-api-access-68d64\") pod \"ingress-canary-ssgsl\" (UID: \"d653889b-b9a3-4d5d-bbf0-f286413a20c9\") " pod="openshift-ingress-canary/ingress-canary-ssgsl"
Jul 28 22:05:38.158641 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.158605    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/331ef64d-2011-4149-b38c-88921f08f078-metrics-tls\") pod \"dns-default-vqx8j\" (UID: \"331ef64d-2011-4149-b38c-88921f08f078\") " pod="openshift-dns/dns-default-vqx8j"
Jul 28 22:05:38.158641 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.158629    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9q28\" (UniqueName: \"kubernetes.io/projected/331ef64d-2011-4149-b38c-88921f08f078-kube-api-access-g9q28\") pod \"dns-default-vqx8j\" (UID: \"331ef64d-2011-4149-b38c-88921f08f078\") " pod="openshift-dns/dns-default-vqx8j"
Jul 28 22:05:38.158798 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.158650    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d653889b-b9a3-4d5d-bbf0-f286413a20c9-cert\") pod \"ingress-canary-ssgsl\" (UID: \"d653889b-b9a3-4d5d-bbf0-f286413a20c9\") " pod="openshift-ingress-canary/ingress-canary-ssgsl"
Jul 28 22:05:38.158798 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.158739    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/331ef64d-2011-4149-b38c-88921f08f078-tmp-dir\") pod \"dns-default-vqx8j\" (UID: \"331ef64d-2011-4149-b38c-88921f08f078\") " pod="openshift-dns/dns-default-vqx8j"
Jul 28 22:05:38.259906 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.259828    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/331ef64d-2011-4149-b38c-88921f08f078-tmp-dir\") pod \"dns-default-vqx8j\" (UID: \"331ef64d-2011-4149-b38c-88921f08f078\") " pod="openshift-dns/dns-default-vqx8j"
Jul 28 22:05:38.259906 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.259890    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/331ef64d-2011-4149-b38c-88921f08f078-config-volume\") pod \"dns-default-vqx8j\" (UID: \"331ef64d-2011-4149-b38c-88921f08f078\") " pod="openshift-dns/dns-default-vqx8j"
Jul 28 22:05:38.260119 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.259928    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68d64\" (UniqueName: \"kubernetes.io/projected/d653889b-b9a3-4d5d-bbf0-f286413a20c9-kube-api-access-68d64\") pod \"ingress-canary-ssgsl\" (UID: \"d653889b-b9a3-4d5d-bbf0-f286413a20c9\") " pod="openshift-ingress-canary/ingress-canary-ssgsl"
Jul 28 22:05:38.260119 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.259972    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/331ef64d-2011-4149-b38c-88921f08f078-metrics-tls\") pod \"dns-default-vqx8j\" (UID: \"331ef64d-2011-4149-b38c-88921f08f078\") " pod="openshift-dns/dns-default-vqx8j"
Jul 28 22:05:38.260119 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.259991    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9q28\" (UniqueName: \"kubernetes.io/projected/331ef64d-2011-4149-b38c-88921f08f078-kube-api-access-g9q28\") pod \"dns-default-vqx8j\" (UID: \"331ef64d-2011-4149-b38c-88921f08f078\") " pod="openshift-dns/dns-default-vqx8j"
Jul 28 22:05:38.260119 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.260014    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d653889b-b9a3-4d5d-bbf0-f286413a20c9-cert\") pod \"ingress-canary-ssgsl\" (UID: \"d653889b-b9a3-4d5d-bbf0-f286413a20c9\") " pod="openshift-ingress-canary/ingress-canary-ssgsl"
Jul 28 22:05:38.260335 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.260300    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/331ef64d-2011-4149-b38c-88921f08f078-tmp-dir\") pod \"dns-default-vqx8j\" (UID: \"331ef64d-2011-4149-b38c-88921f08f078\") " pod="openshift-dns/dns-default-vqx8j"
Jul 28 22:05:38.260427 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:38.260384    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 22:05:38.260483 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:38.260454    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/331ef64d-2011-4149-b38c-88921f08f078-metrics-tls podName:331ef64d-2011-4149-b38c-88921f08f078 nodeName:}" failed. No retries permitted until 2026-07-28 22:05:38.760434523 +0000 UTC m=+36.140572159 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/331ef64d-2011-4149-b38c-88921f08f078-metrics-tls") pod "dns-default-vqx8j" (UID: "331ef64d-2011-4149-b38c-88921f08f078") : secret "dns-default-metrics-tls" not found
Jul 28 22:05:38.260595 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.260568    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/331ef64d-2011-4149-b38c-88921f08f078-config-volume\") pod \"dns-default-vqx8j\" (UID: \"331ef64d-2011-4149-b38c-88921f08f078\") " pod="openshift-dns/dns-default-vqx8j"
Jul 28 22:05:38.264806 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.264655    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d653889b-b9a3-4d5d-bbf0-f286413a20c9-cert\") pod \"ingress-canary-ssgsl\" (UID: \"d653889b-b9a3-4d5d-bbf0-f286413a20c9\") " pod="openshift-ingress-canary/ingress-canary-ssgsl"
Jul 28 22:05:38.269502 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.269454    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9q28\" (UniqueName: \"kubernetes.io/projected/331ef64d-2011-4149-b38c-88921f08f078-kube-api-access-g9q28\") pod \"dns-default-vqx8j\" (UID: \"331ef64d-2011-4149-b38c-88921f08f078\") " pod="openshift-dns/dns-default-vqx8j"
Jul 28 22:05:38.270057 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.270036    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68d64\" (UniqueName: \"kubernetes.io/projected/d653889b-b9a3-4d5d-bbf0-f286413a20c9-kube-api-access-68d64\") pod \"ingress-canary-ssgsl\" (UID: \"d653889b-b9a3-4d5d-bbf0-f286413a20c9\") " pod="openshift-ingress-canary/ingress-canary-ssgsl"
Jul 28 22:05:38.367684 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.367647    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ssgsl"
Jul 28 22:05:38.765166 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:38.765130    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/331ef64d-2011-4149-b38c-88921f08f078-metrics-tls\") pod \"dns-default-vqx8j\" (UID: \"331ef64d-2011-4149-b38c-88921f08f078\") " pod="openshift-dns/dns-default-vqx8j"
Jul 28 22:05:38.765340 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:38.765305    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 22:05:38.765416 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:38.765389    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/331ef64d-2011-4149-b38c-88921f08f078-metrics-tls podName:331ef64d-2011-4149-b38c-88921f08f078 nodeName:}" failed. No retries permitted until 2026-07-28 22:05:39.765368833 +0000 UTC m=+37.145506476 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/331ef64d-2011-4149-b38c-88921f08f078-metrics-tls") pod "dns-default-vqx8j" (UID: "331ef64d-2011-4149-b38c-88921f08f078") : secret "dns-default-metrics-tls" not found
Jul 28 22:05:39.256076 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:39.256044    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ssgsl"]
Jul 28 22:05:39.261665 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:05:39.261642    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd653889b_b9a3_4d5d_bbf0_f286413a20c9.slice/crio-8438c28378687c2ca731ca3f393d67b45f12874d6483fb82c2267430ab9f3f6f WatchSource:0}: Error finding container 8438c28378687c2ca731ca3f393d67b45f12874d6483fb82c2267430ab9f3f6f: Status 404 returned error can't find the container with id 8438c28378687c2ca731ca3f393d67b45f12874d6483fb82c2267430ab9f3f6f
Jul 28 22:05:39.286214 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:39.286183    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ssgsl" event={"ID":"d653889b-b9a3-4d5d-bbf0-f286413a20c9","Type":"ContainerStarted","Data":"8438c28378687c2ca731ca3f393d67b45f12874d6483fb82c2267430ab9f3f6f"}
Jul 28 22:05:39.289160 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:39.289140    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4fjz6" event={"ID":"6eea54c2-c579-4dab-8222-909d88ccf205","Type":"ContainerStarted","Data":"4f56a25177cb424c3f792d5938c648500da7b3c10645e6b0bf7a28bce8b2601b"}
Jul 28 22:05:39.774642 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:39.774600    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/331ef64d-2011-4149-b38c-88921f08f078-metrics-tls\") pod \"dns-default-vqx8j\" (UID: \"331ef64d-2011-4149-b38c-88921f08f078\") " pod="openshift-dns/dns-default-vqx8j"
Jul 28 22:05:39.774809 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:39.774754    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 22:05:39.774849 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:39.774835    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/331ef64d-2011-4149-b38c-88921f08f078-metrics-tls podName:331ef64d-2011-4149-b38c-88921f08f078 nodeName:}" failed. No retries permitted until 2026-07-28 22:05:41.774817346 +0000 UTC m=+39.154954968 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/331ef64d-2011-4149-b38c-88921f08f078-metrics-tls") pod "dns-default-vqx8j" (UID: "331ef64d-2011-4149-b38c-88921f08f078") : secret "dns-default-metrics-tls" not found
Jul 28 22:05:40.294420 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:40.294381    2563 generic.go:356] "Generic (PLEG): container finished" podID="6eea54c2-c579-4dab-8222-909d88ccf205" containerID="4f56a25177cb424c3f792d5938c648500da7b3c10645e6b0bf7a28bce8b2601b" exitCode=0
Jul 28 22:05:40.294871 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:40.294436    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4fjz6" event={"ID":"6eea54c2-c579-4dab-8222-909d88ccf205","Type":"ContainerDied","Data":"4f56a25177cb424c3f792d5938c648500da7b3c10645e6b0bf7a28bce8b2601b"}
Jul 28 22:05:41.300524 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:41.300277    2563 generic.go:356] "Generic (PLEG): container finished" podID="6eea54c2-c579-4dab-8222-909d88ccf205" containerID="a90ac2dbdcfaea791da39088e52ee19d2d7a630377d27ccadb6cf650ca03181e" exitCode=0
Jul 28 22:05:41.300906 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:41.300374    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4fjz6" event={"ID":"6eea54c2-c579-4dab-8222-909d88ccf205","Type":"ContainerDied","Data":"a90ac2dbdcfaea791da39088e52ee19d2d7a630377d27ccadb6cf650ca03181e"}
Jul 28 22:05:41.791094 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:41.791070    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/331ef64d-2011-4149-b38c-88921f08f078-metrics-tls\") pod \"dns-default-vqx8j\" (UID: \"331ef64d-2011-4149-b38c-88921f08f078\") " pod="openshift-dns/dns-default-vqx8j"
Jul 28 22:05:41.791222 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:41.791202    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 22:05:41.791270 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:41.791261    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/331ef64d-2011-4149-b38c-88921f08f078-metrics-tls podName:331ef64d-2011-4149-b38c-88921f08f078 nodeName:}" failed. No retries permitted until 2026-07-28 22:05:45.791244302 +0000 UTC m=+43.171381924 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/331ef64d-2011-4149-b38c-88921f08f078-metrics-tls") pod "dns-default-vqx8j" (UID: "331ef64d-2011-4149-b38c-88921f08f078") : secret "dns-default-metrics-tls" not found
Jul 28 22:05:42.306369 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:42.306326    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4fjz6" event={"ID":"6eea54c2-c579-4dab-8222-909d88ccf205","Type":"ContainerStarted","Data":"fb2fa6893f420854d7c0248f5010543359cc56085dd1982882557ac9402cad90"}
Jul 28 22:05:42.307693 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:42.307672    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ssgsl" event={"ID":"d653889b-b9a3-4d5d-bbf0-f286413a20c9","Type":"ContainerStarted","Data":"1176adeafc00fc0a04128efa4e866f43904545835b9f0a798eaa2f5e64461b3c"}
Jul 28 22:05:42.334576 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:42.334523    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4fjz6" podStartSLOduration=8.191195108 podStartE2EDuration="38.33450799s" totalImagesPullingTime="30.143312882s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:05:04 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:05:42.333139925 +0000 UTC m=+39.713277568" watchObservedRunningTime="2026-07-28 22:05:42.33450799 +0000 UTC m=+39.714645634"
Jul 28 22:05:42.353211 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:42.350138    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ssgsl" podStartSLOduration=1.810943792 podStartE2EDuration="4.350122144s" totalImagesPullingTime="2.539178352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:05:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:05:42.348840816 +0000 UTC m=+39.728978459" watchObservedRunningTime="2026-07-28 22:05:42.350122144 +0000 UTC m=+39.730259767"
Jul 28 22:05:45.816811 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:45.816773    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/331ef64d-2011-4149-b38c-88921f08f078-metrics-tls\") pod \"dns-default-vqx8j\" (UID: \"331ef64d-2011-4149-b38c-88921f08f078\") " pod="openshift-dns/dns-default-vqx8j"
Jul 28 22:05:45.817185 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:45.816901    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 22:05:45.817185 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:45.816955    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/331ef64d-2011-4149-b38c-88921f08f078-metrics-tls podName:331ef64d-2011-4149-b38c-88921f08f078 nodeName:}" failed. No retries permitted until 2026-07-28 22:05:53.81694118 +0000 UTC m=+51.197078803 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/331ef64d-2011-4149-b38c-88921f08f078-metrics-tls") pod "dns-default-vqx8j" (UID: "331ef64d-2011-4149-b38c-88921f08f078") : secret "dns-default-metrics-tls" not found
Jul 28 22:05:53.871465 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:05:53.871421    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/331ef64d-2011-4149-b38c-88921f08f078-metrics-tls\") pod \"dns-default-vqx8j\" (UID: \"331ef64d-2011-4149-b38c-88921f08f078\") " pod="openshift-dns/dns-default-vqx8j"
Jul 28 22:05:53.871929 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:53.871549    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 22:05:53.871929 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:05:53.871620    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/331ef64d-2011-4149-b38c-88921f08f078-metrics-tls podName:331ef64d-2011-4149-b38c-88921f08f078 nodeName:}" failed. No retries permitted until 2026-07-28 22:06:09.871602638 +0000 UTC m=+67.251740280 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/331ef64d-2011-4149-b38c-88921f08f078-metrics-tls") pod "dns-default-vqx8j" (UID: "331ef64d-2011-4149-b38c-88921f08f078") : secret "dns-default-metrics-tls" not found
Jul 28 22:06:03.284404 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:03.284376    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mcj8p"
Jul 28 22:06:08.771468 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:08.771428    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d7dfdb2-a77a-47e8-865a-07a014ec5633-original-pull-secret\") pod \"global-pull-secret-syncer-2rqhg\" (UID: \"5d7dfdb2-a77a-47e8-865a-07a014ec5633\") " pod="kube-system/global-pull-secret-syncer-2rqhg"
Jul 28 22:06:08.774022 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:08.774002    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 22:06:08.785340 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:08.785316    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d7dfdb2-a77a-47e8-865a-07a014ec5633-original-pull-secret\") pod \"global-pull-secret-syncer-2rqhg\" (UID: \"5d7dfdb2-a77a-47e8-865a-07a014ec5633\") " pod="kube-system/global-pull-secret-syncer-2rqhg"
Jul 28 22:06:08.872729 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:08.872686    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f33e531-3341-4c01-940e-91f17519976c-metrics-certs\") pod \"network-metrics-daemon-4xqhf\" (UID: \"2f33e531-3341-4c01-940e-91f17519976c\") " pod="openshift-multus/network-metrics-daemon-4xqhf"
Jul 28 22:06:08.875292 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:08.875276    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 22:06:08.883406 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:06:08.883383    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 28 22:06:08.883484 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:06:08.883454    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f33e531-3341-4c01-940e-91f17519976c-metrics-certs podName:2f33e531-3341-4c01-940e-91f17519976c nodeName:}" failed. No retries permitted until 2026-07-28 22:07:12.883431689 +0000 UTC m=+130.263569313 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f33e531-3341-4c01-940e-91f17519976c-metrics-certs") pod "network-metrics-daemon-4xqhf" (UID: "2f33e531-3341-4c01-940e-91f17519976c") : secret "metrics-daemon-secret" not found
Jul 28 22:06:08.973949 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:08.973896    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntdgg\" (UniqueName: \"kubernetes.io/projected/962b24a3-abd7-4a45-98fa-f225c383b0dc-kube-api-access-ntdgg\") pod \"network-check-target-87zr2\" (UID: \"962b24a3-abd7-4a45-98fa-f225c383b0dc\") " pod="openshift-network-diagnostics/network-check-target-87zr2"
Jul 28 22:06:08.976451 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:08.976434    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 22:06:08.986150 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:08.986131    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 22:06:08.997284 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:08.997263    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntdgg\" (UniqueName: \"kubernetes.io/projected/962b24a3-abd7-4a45-98fa-f225c383b0dc-kube-api-access-ntdgg\") pod \"network-check-target-87zr2\" (UID: \"962b24a3-abd7-4a45-98fa-f225c383b0dc\") " pod="openshift-network-diagnostics/network-check-target-87zr2"
Jul 28 22:06:09.043882 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:09.043821    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rqhg"
Jul 28 22:06:09.052991 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:09.052968    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-65j7n\""
Jul 28 22:06:09.061255 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:09.061230    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-87zr2"
Jul 28 22:06:09.168473 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:09.168443    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2rqhg"]
Jul 28 22:06:09.184154 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:09.184132    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-87zr2"]
Jul 28 22:06:09.187446 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:06:09.187426    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod962b24a3_abd7_4a45_98fa_f225c383b0dc.slice/crio-c9aacd3605de0b87535d1177d3c4ddac0d8c5a2480dbcfe36be2f986a56fdd23 WatchSource:0}: Error finding container c9aacd3605de0b87535d1177d3c4ddac0d8c5a2480dbcfe36be2f986a56fdd23: Status 404 returned error can't find the container with id c9aacd3605de0b87535d1177d3c4ddac0d8c5a2480dbcfe36be2f986a56fdd23
Jul 28 22:06:09.368940 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:09.368861    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-87zr2" event={"ID":"962b24a3-abd7-4a45-98fa-f225c383b0dc","Type":"ContainerStarted","Data":"c9aacd3605de0b87535d1177d3c4ddac0d8c5a2480dbcfe36be2f986a56fdd23"}
Jul 28 22:06:09.369915 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:09.369890    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2rqhg" event={"ID":"5d7dfdb2-a77a-47e8-865a-07a014ec5633","Type":"ContainerStarted","Data":"57faa3969a9498c6254f979f296f9d0ec4d4fa59c6742e534c2604612750f3b3"}
Jul 28 22:06:09.880098 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:09.880058    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/331ef64d-2011-4149-b38c-88921f08f078-metrics-tls\") pod \"dns-default-vqx8j\" (UID: \"331ef64d-2011-4149-b38c-88921f08f078\") " pod="openshift-dns/dns-default-vqx8j"
Jul 28 22:06:09.880649 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:06:09.880208    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 22:06:09.880649 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:06:09.880278    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/331ef64d-2011-4149-b38c-88921f08f078-metrics-tls podName:331ef64d-2011-4149-b38c-88921f08f078 nodeName:}" failed. No retries permitted until 2026-07-28 22:06:41.880257667 +0000 UTC m=+99.260395290 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/331ef64d-2011-4149-b38c-88921f08f078-metrics-tls") pod "dns-default-vqx8j" (UID: "331ef64d-2011-4149-b38c-88921f08f078") : secret "dns-default-metrics-tls" not found
Jul 28 22:06:14.383807 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:14.383767    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-87zr2" event={"ID":"962b24a3-abd7-4a45-98fa-f225c383b0dc","Type":"ContainerStarted","Data":"6cedd0c4d5ba3645c6fbb29803703bf424bec6c400bea02dd7d632cea0668233"}
Jul 28 22:06:14.384258 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:14.383867    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-87zr2"
Jul 28 22:06:14.385161 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:14.385138    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2rqhg" event={"ID":"5d7dfdb2-a77a-47e8-865a-07a014ec5633","Type":"ContainerStarted","Data":"206d0ffc058f3bb3917dbf26d0b6c7f343afd91b1eb5bb6e1e9e9ebd1f894554"}
Jul 28 22:06:14.399134 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:14.399086    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-87zr2" podStartSLOduration=65.877640214 podStartE2EDuration="1m10.399075047s" totalImagesPullingTime="4.521434833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:05:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:06:14.398847506 +0000 UTC m=+71.778985149" watchObservedRunningTime="2026-07-28 22:06:14.399075047 +0000 UTC m=+71.779212690"
Jul 28 22:06:14.414019 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:14.413976    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2rqhg" podStartSLOduration=65.875300112 podStartE2EDuration="1m10.413963562s" totalImagesPullingTime="4.53866345s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:05:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:06:14.412758708 +0000 UTC m=+71.792896352" watchObservedRunningTime="2026-07-28 22:06:14.413963562 +0000 UTC m=+71.794101206"
Jul 28 22:06:34.174025 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:34.173890    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-557bbf488c-ls2gh"]
Jul 28 22:06:34.176587 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:34.176564    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-557bbf488c-ls2gh"
Jul 28 22:06:34.181907 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:34.181861    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 22:06:34.182050 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:34.181919    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 22:06:34.182050 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:34.181926    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 22:06:34.182050 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:34.181924    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 28 22:06:34.193865 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:34.193844    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-557bbf488c-ls2gh"]
Jul 28 22:06:34.238111 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:34.238077    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-799ff6cbf6-5vlzx"]
Jul 28 22:06:34.240048 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:34.240030    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-799ff6cbf6-5vlzx"
Jul 28 22:06:34.244089 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:34.244071    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 28 22:06:34.244207 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:34.244191    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-n6pdt\""
Jul 28 22:06:34.246230 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:34.246211    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bad2568e-47bd-40d6-ae05-5020535421d4-tmp\") pod \"klusterlet-addon-workmgr-557bbf488c-ls2gh\" (UID: \"bad2568e-47bd-40d6-ae05-5020535421d4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-557bbf488c-ls2gh"
Jul 28 22:06:34.246332 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:34.246308    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bad2568e-47bd-40d6-ae05-5020535421d4-klusterlet-config\") pod \"klusterlet-addon-workmgr-557bbf488c-ls2gh\" (UID: \"bad2568e-47bd-40d6-ae05-5020535421d4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-557bbf488c-ls2gh"
Jul 28 22:06:34.246417 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:34.246338    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkcsl\" (UniqueName: \"kubernetes.io/projected/bad2568e-47bd-40d6-ae05-5020535421d4-kube-api-access-dkcsl\") pod \"klusterlet-addon-workmgr-557bbf488c-ls2gh\" (UID: \"bad2568e-47bd-40d6-ae05-5020535421d4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-557bbf488c-ls2gh"
Jul 28 22:06:34.254059 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:34.254033    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-799ff6cbf6-5vlzx"]
Jul 28 22:06:34.347541 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:34.347508    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bad2568e-47bd-40d6-ae05-5020535421d4-tmp\") pod \"klusterlet-addon-workmgr-557bbf488c-ls2gh\" (UID: \"bad2568e-47bd-40d6-ae05-5020535421d4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-557bbf488c-ls2gh"
Jul 28 22:06:34.347694 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:34.347560    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4fw7\" (UniqueName: \"kubernetes.io/projected/9fdb31f0-ee18-44db-ad18-f5e0f29f3add-kube-api-access-s4fw7\") pod \"managed-serviceaccount-addon-agent-799ff6cbf6-5vlzx\" (UID: \"9fdb31f0-ee18-44db-ad18-f5e0f29f3add\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-799ff6cbf6-5vlzx"
Jul 28 22:06:34.347738 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:34.347713    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bad2568e-47bd-40d6-ae05-5020535421d4-klusterlet-config\") pod \"klusterlet-addon-workmgr-557bbf488c-ls2gh\" (UID: \"bad2568e-47bd-40d6-ae05-5020535421d4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-557bbf488c-ls2gh"
Jul 28 22:06:34.347776 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:34.347750    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkcsl\" (UniqueName: \"kubernetes.io/projected/bad2568e-47bd-40d6-ae05-5020535421d4-kube-api-access-dkcsl\") pod \"klusterlet-addon-workmgr-557bbf488c-ls2gh\" (UID: \"bad2568e-47bd-40d6-ae05-5020535421d4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-557bbf488c-ls2gh"
Jul 28 22:06:34.347817 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:34.347783    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9fdb31f0-ee18-44db-ad18-f5e0f29f3add-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-799ff6cbf6-5vlzx\" (UID: \"9fdb31f0-ee18-44db-ad18-f5e0f29f3add\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-799ff6cbf6-5vlzx"
Jul 28 22:06:34.347916 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:34.347902    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bad2568e-47bd-40d6-ae05-5020535421d4-tmp\") pod \"klusterlet-addon-workmgr-557bbf488c-ls2gh\" (UID: \"bad2568e-47bd-40d6-ae05-5020535421d4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-557bbf488c-ls2gh"
Jul 28 22:06:34.350078 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:34.350052    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bad2568e-47bd-40d6-ae05-5020535421d4-klusterlet-config\") pod \"klusterlet-addon-workmgr-557bbf488c-ls2gh\" (UID: \"bad2568e-47bd-40d6-ae05-5020535421d4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-557bbf488c-ls2gh"
Jul 28 22:06:34.357161 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:34.357139    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkcsl\" (UniqueName: \"kubernetes.io/projected/bad2568e-47bd-40d6-ae05-5020535421d4-kube-api-access-dkcsl\") pod \"klusterlet-addon-workmgr-557bbf488c-ls2gh\" (UID: \"bad2568e-47bd-40d6-ae05-5020535421d4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-557bbf488c-ls2gh"
Jul 28 22:06:34.448173 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:34.448150    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4fw7\" (UniqueName: \"kubernetes.io/projected/9fdb31f0-ee18-44db-ad18-f5e0f29f3add-kube-api-access-s4fw7\") pod \"managed-serviceaccount-addon-agent-799ff6cbf6-5vlzx\" (UID: \"9fdb31f0-ee18-44db-ad18-f5e0f29f3add\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-799ff6cbf6-5vlzx"
Jul 28 22:06:34.448286 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:34.448196    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9fdb31f0-ee18-44db-ad18-f5e0f29f3add-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-799ff6cbf6-5vlzx\" (UID: \"9fdb31f0-ee18-44db-ad18-f5e0f29f3add\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-799ff6cbf6-5vlzx"
Jul 28 22:06:34.450386 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:34.450368    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9fdb31f0-ee18-44db-ad18-f5e0f29f3add-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-799ff6cbf6-5vlzx\" (UID: \"9fdb31f0-ee18-44db-ad18-f5e0f29f3add\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-799ff6cbf6-5vlzx"
Jul 28 22:06:34.459735 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:34.459705    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4fw7\" (UniqueName: \"kubernetes.io/projected/9fdb31f0-ee18-44db-ad18-f5e0f29f3add-kube-api-access-s4fw7\") pod \"managed-serviceaccount-addon-agent-799ff6cbf6-5vlzx\" (UID: \"9fdb31f0-ee18-44db-ad18-f5e0f29f3add\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-799ff6cbf6-5vlzx"
Jul 28 22:06:34.484535 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:34.484516    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-557bbf488c-ls2gh"
Jul 28 22:06:34.562681 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:34.562648    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-799ff6cbf6-5vlzx"
Jul 28 22:06:34.624793 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:34.624768    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-557bbf488c-ls2gh"]
Jul 28 22:06:34.626752 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:06:34.626721    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbad2568e_47bd_40d6_ae05_5020535421d4.slice/crio-198c76866e17b0e98e0712b0d3fff7c361b308bed17a7c2f86e96a3e1996182a WatchSource:0}: Error finding container 198c76866e17b0e98e0712b0d3fff7c361b308bed17a7c2f86e96a3e1996182a: Status 404 returned error can't find the container with id 198c76866e17b0e98e0712b0d3fff7c361b308bed17a7c2f86e96a3e1996182a
Jul 28 22:06:34.698586 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:34.698516    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-799ff6cbf6-5vlzx"]
Jul 28 22:06:34.701126 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:06:34.701104    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9fdb31f0_ee18_44db_ad18_f5e0f29f3add.slice/crio-e9cffa408eb894e411db9dcb956680aeff5a9e4e8905b7dd21f3dd1c9b95ab01 WatchSource:0}: Error finding container e9cffa408eb894e411db9dcb956680aeff5a9e4e8905b7dd21f3dd1c9b95ab01: Status 404 returned error can't find the container with id e9cffa408eb894e411db9dcb956680aeff5a9e4e8905b7dd21f3dd1c9b95ab01
Jul 28 22:06:35.434752 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:35.434720    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-799ff6cbf6-5vlzx" event={"ID":"9fdb31f0-ee18-44db-ad18-f5e0f29f3add","Type":"ContainerStarted","Data":"e9cffa408eb894e411db9dcb956680aeff5a9e4e8905b7dd21f3dd1c9b95ab01"}
Jul 28 22:06:35.435821 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:35.435801    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-557bbf488c-ls2gh" event={"ID":"bad2568e-47bd-40d6-ae05-5020535421d4","Type":"ContainerStarted","Data":"198c76866e17b0e98e0712b0d3fff7c361b308bed17a7c2f86e96a3e1996182a"}
Jul 28 22:06:38.447806 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:38.447764    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-799ff6cbf6-5vlzx" event={"ID":"9fdb31f0-ee18-44db-ad18-f5e0f29f3add","Type":"ContainerStarted","Data":"9bcd7644bee9893d5b6d369dfcf1f113ef952e11402468fa67823f06c21adfac"}
Jul 28 22:06:38.475004 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:38.474925    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-799ff6cbf6-5vlzx" podStartSLOduration=1.252101921 podStartE2EDuration="4.474903801s" totalImagesPullingTime="3.22280188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:06:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:06:38.473393265 +0000 UTC m=+95.853530909" watchObservedRunningTime="2026-07-28 22:06:38.474903801 +0000 UTC m=+95.855041445"
Jul 28 22:06:39.451741 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:39.451705    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-557bbf488c-ls2gh" event={"ID":"bad2568e-47bd-40d6-ae05-5020535421d4","Type":"ContainerStarted","Data":"6ef452d10c98cd7130a91e8b771619c8cd6126d1c67bf75ba3d4c5b4751b5cb9"}
Jul 28 22:06:39.468724 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:39.468652    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-557bbf488c-ls2gh" podStartSLOduration=0.918576694 podStartE2EDuration="5.468637462s" totalImagesPullingTime="4.550060768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:06:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:06:39.467783185 +0000 UTC m=+96.847920831" watchObservedRunningTime="2026-07-28 22:06:39.468637462 +0000 UTC m=+96.848775154"
Jul 28 22:06:40.454048 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:40.454018    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-557bbf488c-ls2gh"
Jul 28 22:06:40.455602 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:40.455580    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-557bbf488c-ls2gh"
Jul 28 22:06:41.903682 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:41.903647    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/331ef64d-2011-4149-b38c-88921f08f078-metrics-tls\") pod \"dns-default-vqx8j\" (UID: \"331ef64d-2011-4149-b38c-88921f08f078\") " pod="openshift-dns/dns-default-vqx8j"
Jul 28 22:06:41.904067 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:06:41.903768    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 22:06:41.904067 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:06:41.903821    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/331ef64d-2011-4149-b38c-88921f08f078-metrics-tls podName:331ef64d-2011-4149-b38c-88921f08f078 nodeName:}" failed. No retries permitted until 2026-07-28 22:07:45.903805613 +0000 UTC m=+163.283943234 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/331ef64d-2011-4149-b38c-88921f08f078-metrics-tls") pod "dns-default-vqx8j" (UID: "331ef64d-2011-4149-b38c-88921f08f078") : secret "dns-default-metrics-tls" not found
Jul 28 22:06:45.390338 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:06:45.390309    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-87zr2"
Jul 28 22:07:12.924054 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:07:12.923999    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f33e531-3341-4c01-940e-91f17519976c-metrics-certs\") pod \"network-metrics-daemon-4xqhf\" (UID: \"2f33e531-3341-4c01-940e-91f17519976c\") " pod="openshift-multus/network-metrics-daemon-4xqhf"
Jul 28 22:07:12.924645 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:07:12.924172    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 28 22:07:12.924645 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:07:12.924255    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f33e531-3341-4c01-940e-91f17519976c-metrics-certs podName:2f33e531-3341-4c01-940e-91f17519976c nodeName:}" failed. No retries permitted until 2026-07-28 22:09:14.924233398 +0000 UTC m=+252.304371036 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f33e531-3341-4c01-940e-91f17519976c-metrics-certs") pod "network-metrics-daemon-4xqhf" (UID: "2f33e531-3341-4c01-940e-91f17519976c") : secret "metrics-daemon-secret" not found
Jul 28 22:07:41.061500 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:07:41.061448    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-vqx8j" podUID="331ef64d-2011-4149-b38c-88921f08f078"
Jul 28 22:07:41.136481 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:07:41.136449    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-4xqhf" podUID="2f33e531-3341-4c01-940e-91f17519976c"
Jul 28 22:07:41.625640 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:07:41.625606    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vqx8j"
Jul 28 22:07:45.929443 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:07:45.929397    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/331ef64d-2011-4149-b38c-88921f08f078-metrics-tls\") pod \"dns-default-vqx8j\" (UID: \"331ef64d-2011-4149-b38c-88921f08f078\") " pod="openshift-dns/dns-default-vqx8j"
Jul 28 22:07:45.929877 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:07:45.929557    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 22:07:45.929877 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:07:45.929628    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/331ef64d-2011-4149-b38c-88921f08f078-metrics-tls podName:331ef64d-2011-4149-b38c-88921f08f078 nodeName:}" failed. No retries permitted until 2026-07-28 22:09:47.929610122 +0000 UTC m=+285.309747746 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/331ef64d-2011-4149-b38c-88921f08f078-metrics-tls") pod "dns-default-vqx8j" (UID: "331ef64d-2011-4149-b38c-88921f08f078") : secret "dns-default-metrics-tls" not found
Jul 28 22:07:53.125613 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:07:53.125579    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4xqhf"
Jul 28 22:08:38.776996 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:08:38.776965    2563 generic.go:356] "Generic (PLEG): container finished" podID="9fdb31f0-ee18-44db-ad18-f5e0f29f3add" containerID="9bcd7644bee9893d5b6d369dfcf1f113ef952e11402468fa67823f06c21adfac" exitCode=255
Jul 28 22:08:38.777426 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:08:38.777018    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-799ff6cbf6-5vlzx" event={"ID":"9fdb31f0-ee18-44db-ad18-f5e0f29f3add","Type":"ContainerDied","Data":"9bcd7644bee9893d5b6d369dfcf1f113ef952e11402468fa67823f06c21adfac"}
Jul 28 22:08:38.782516 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:08:38.782500    2563 scope.go:117] "RemoveContainer" containerID="9bcd7644bee9893d5b6d369dfcf1f113ef952e11402468fa67823f06c21adfac"
Jul 28 22:08:39.782184 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:08:39.782151    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-799ff6cbf6-5vlzx" event={"ID":"9fdb31f0-ee18-44db-ad18-f5e0f29f3add","Type":"ContainerStarted","Data":"d2765fc09db0b39d73a608391b0e712bf6b7eff7911cd1f47198c77163f1deb3"}
Jul 28 22:08:39.783497 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:08:39.783474    2563 generic.go:356] "Generic (PLEG): container finished" podID="bad2568e-47bd-40d6-ae05-5020535421d4" containerID="6ef452d10c98cd7130a91e8b771619c8cd6126d1c67bf75ba3d4c5b4751b5cb9" exitCode=1
Jul 28 22:08:39.783601 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:08:39.783522    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-557bbf488c-ls2gh" event={"ID":"bad2568e-47bd-40d6-ae05-5020535421d4","Type":"ContainerDied","Data":"6ef452d10c98cd7130a91e8b771619c8cd6126d1c67bf75ba3d4c5b4751b5cb9"}
Jul 28 22:08:39.783763 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:08:39.783752    2563 scope.go:117] "RemoveContainer" containerID="6ef452d10c98cd7130a91e8b771619c8cd6126d1c67bf75ba3d4c5b4751b5cb9"
Jul 28 22:08:40.455007 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:08:40.454967    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-557bbf488c-ls2gh"
Jul 28 22:08:40.788211 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:08:40.788132    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-557bbf488c-ls2gh" event={"ID":"bad2568e-47bd-40d6-ae05-5020535421d4","Type":"ContainerStarted","Data":"b5fe9e615ef95cfc70e80227245f0828bb823fb8b6795c6c77e2e89b4fbd9fd5"}
Jul 28 22:08:40.788647 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:08:40.788322    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-557bbf488c-ls2gh"
Jul 28 22:08:40.788924 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:08:40.788906    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-557bbf488c-ls2gh"
Jul 28 22:09:01.304411 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:01.304379    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f9c64b766-fwkvf"]
Jul 28 22:09:01.306335 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:01.306318    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf"
Jul 28 22:09:01.309315 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:01.309280    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 22:09:01.309315 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:01.309295    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 22:09:01.309494 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:01.309321    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8lc2p\""
Jul 28 22:09:01.309494 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:01.309299    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 22:09:01.315367 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:01.315339    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 22:09:01.336816 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:01.336793    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f9c64b766-fwkvf"]
Jul 28 22:09:01.436964 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:01.436935    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/edcb52a8-15ce-419d-93dc-cc405f75076e-ca-trust-extracted\") pod \"image-registry-7f9c64b766-fwkvf\" (UID: \"edcb52a8-15ce-419d-93dc-cc405f75076e\") " pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf"
Jul 28 22:09:01.437151 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:01.436976    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdsjj\" (UniqueName: \"kubernetes.io/projected/edcb52a8-15ce-419d-93dc-cc405f75076e-kube-api-access-hdsjj\") pod \"image-registry-7f9c64b766-fwkvf\" (UID: \"edcb52a8-15ce-419d-93dc-cc405f75076e\") " pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf"
Jul 28 22:09:01.437151 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:01.437007    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/edcb52a8-15ce-419d-93dc-cc405f75076e-registry-tls\") pod \"image-registry-7f9c64b766-fwkvf\" (UID: \"edcb52a8-15ce-419d-93dc-cc405f75076e\") " pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf"
Jul 28 22:09:01.437151 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:01.437030    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/edcb52a8-15ce-419d-93dc-cc405f75076e-image-registry-private-configuration\") pod \"image-registry-7f9c64b766-fwkvf\" (UID: \"edcb52a8-15ce-419d-93dc-cc405f75076e\") " pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf"
Jul 28 22:09:01.437151 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:01.437068    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/edcb52a8-15ce-419d-93dc-cc405f75076e-registry-certificates\") pod \"image-registry-7f9c64b766-fwkvf\" (UID: \"edcb52a8-15ce-419d-93dc-cc405f75076e\") " pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf"
Jul 28 22:09:01.437151 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:01.437106    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/edcb52a8-15ce-419d-93dc-cc405f75076e-trusted-ca\") pod \"image-registry-7f9c64b766-fwkvf\" (UID: \"edcb52a8-15ce-419d-93dc-cc405f75076e\") " pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf"
Jul 28 22:09:01.437151 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:01.437154    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/edcb52a8-15ce-419d-93dc-cc405f75076e-bound-sa-token\") pod \"image-registry-7f9c64b766-fwkvf\" (UID: \"edcb52a8-15ce-419d-93dc-cc405f75076e\") " pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf"
Jul 28 22:09:01.437346 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:01.437171    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/edcb52a8-15ce-419d-93dc-cc405f75076e-installation-pull-secrets\") pod \"image-registry-7f9c64b766-fwkvf\" (UID: \"edcb52a8-15ce-419d-93dc-cc405f75076e\") " pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf"
Jul 28 22:09:01.537483 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:01.537445    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/edcb52a8-15ce-419d-93dc-cc405f75076e-ca-trust-extracted\") pod \"image-registry-7f9c64b766-fwkvf\" (UID: \"edcb52a8-15ce-419d-93dc-cc405f75076e\") " pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf"
Jul 28 22:09:01.537483 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:01.537489    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdsjj\" (UniqueName: \"kubernetes.io/projected/edcb52a8-15ce-419d-93dc-cc405f75076e-kube-api-access-hdsjj\") pod \"image-registry-7f9c64b766-fwkvf\" (UID: \"edcb52a8-15ce-419d-93dc-cc405f75076e\") " pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf"
Jul 28 22:09:01.537729 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:01.537515    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/edcb52a8-15ce-419d-93dc-cc405f75076e-registry-tls\") pod \"image-registry-7f9c64b766-fwkvf\" (UID: \"edcb52a8-15ce-419d-93dc-cc405f75076e\") " pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf"
Jul 28 22:09:01.537729 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:01.537539    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/edcb52a8-15ce-419d-93dc-cc405f75076e-image-registry-private-configuration\") pod \"image-registry-7f9c64b766-fwkvf\" (UID: \"edcb52a8-15ce-419d-93dc-cc405f75076e\") " pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf"
Jul 28 22:09:01.537729 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:01.537566    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/edcb52a8-15ce-419d-93dc-cc405f75076e-registry-certificates\") pod \"image-registry-7f9c64b766-fwkvf\" (UID: \"edcb52a8-15ce-419d-93dc-cc405f75076e\") " pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf"
Jul 28 22:09:01.537729 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:01.537590    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/edcb52a8-15ce-419d-93dc-cc405f75076e-trusted-ca\") pod \"image-registry-7f9c64b766-fwkvf\" (UID: \"edcb52a8-15ce-419d-93dc-cc405f75076e\") " pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf"
Jul 28 22:09:01.537729 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:01.537617    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/edcb52a8-15ce-419d-93dc-cc405f75076e-bound-sa-token\") pod \"image-registry-7f9c64b766-fwkvf\" (UID: \"edcb52a8-15ce-419d-93dc-cc405f75076e\") " pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf"
Jul 28 22:09:01.537729 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:01.537637    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 22:09:01.537729 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:01.537655    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f9c64b766-fwkvf: secret "image-registry-tls" not found
Jul 28 22:09:01.538042 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:01.537641    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/edcb52a8-15ce-419d-93dc-cc405f75076e-installation-pull-secrets\") pod \"image-registry-7f9c64b766-fwkvf\" (UID: \"edcb52a8-15ce-419d-93dc-cc405f75076e\") " pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf"
Jul 28 22:09:01.538042 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:01.537766    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edcb52a8-15ce-419d-93dc-cc405f75076e-registry-tls podName:edcb52a8-15ce-419d-93dc-cc405f75076e nodeName:}" failed. No retries permitted until 2026-07-28 22:09:02.037745428 +0000 UTC m=+239.417883056 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/edcb52a8-15ce-419d-93dc-cc405f75076e-registry-tls") pod "image-registry-7f9c64b766-fwkvf" (UID: "edcb52a8-15ce-419d-93dc-cc405f75076e") : secret "image-registry-tls" not found
Jul 28 22:09:01.538042 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:01.537943    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/edcb52a8-15ce-419d-93dc-cc405f75076e-ca-trust-extracted\") pod \"image-registry-7f9c64b766-fwkvf\" (UID: \"edcb52a8-15ce-419d-93dc-cc405f75076e\") " pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf"
Jul 28 22:09:01.538263 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:01.538240    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/edcb52a8-15ce-419d-93dc-cc405f75076e-registry-certificates\") pod \"image-registry-7f9c64b766-fwkvf\" (UID: \"edcb52a8-15ce-419d-93dc-cc405f75076e\") " pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf"
Jul 28 22:09:01.538530 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:01.538511    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/edcb52a8-15ce-419d-93dc-cc405f75076e-trusted-ca\") pod \"image-registry-7f9c64b766-fwkvf\" (UID: \"edcb52a8-15ce-419d-93dc-cc405f75076e\") " pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf"
Jul 28 22:09:01.539893 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:01.539872    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/edcb52a8-15ce-419d-93dc-cc405f75076e-installation-pull-secrets\") pod \"image-registry-7f9c64b766-fwkvf\" (UID: \"edcb52a8-15ce-419d-93dc-cc405f75076e\") " pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf"
Jul 28 22:09:01.540018 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:01.540002    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/edcb52a8-15ce-419d-93dc-cc405f75076e-image-registry-private-configuration\") pod \"image-registry-7f9c64b766-fwkvf\" (UID: \"edcb52a8-15ce-419d-93dc-cc405f75076e\") " pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf"
Jul 28 22:09:01.547114 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:01.547093    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/edcb52a8-15ce-419d-93dc-cc405f75076e-bound-sa-token\") pod \"image-registry-7f9c64b766-fwkvf\" (UID: \"edcb52a8-15ce-419d-93dc-cc405f75076e\") " pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf"
Jul 28 22:09:01.547422 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:01.547403    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdsjj\" (UniqueName: \"kubernetes.io/projected/edcb52a8-15ce-419d-93dc-cc405f75076e-kube-api-access-hdsjj\") pod \"image-registry-7f9c64b766-fwkvf\" (UID: \"edcb52a8-15ce-419d-93dc-cc405f75076e\") " pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf"
Jul 28 22:09:02.040980 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:02.040931    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/edcb52a8-15ce-419d-93dc-cc405f75076e-registry-tls\") pod \"image-registry-7f9c64b766-fwkvf\" (UID: \"edcb52a8-15ce-419d-93dc-cc405f75076e\") " pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf"
Jul 28 22:09:02.041163 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:02.041085    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 22:09:02.041163 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:02.041105    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f9c64b766-fwkvf: secret "image-registry-tls" not found
Jul 28 22:09:02.041255 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:02.041169    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edcb52a8-15ce-419d-93dc-cc405f75076e-registry-tls podName:edcb52a8-15ce-419d-93dc-cc405f75076e nodeName:}" failed. No retries permitted until 2026-07-28 22:09:03.041153937 +0000 UTC m=+240.421291562 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/edcb52a8-15ce-419d-93dc-cc405f75076e-registry-tls") pod "image-registry-7f9c64b766-fwkvf" (UID: "edcb52a8-15ce-419d-93dc-cc405f75076e") : secret "image-registry-tls" not found
Jul 28 22:09:03.048245 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:03.048215    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/edcb52a8-15ce-419d-93dc-cc405f75076e-registry-tls\") pod \"image-registry-7f9c64b766-fwkvf\" (UID: \"edcb52a8-15ce-419d-93dc-cc405f75076e\") " pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf"
Jul 28 22:09:03.048680 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:03.048308    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 22:09:03.048680 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:03.048319    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f9c64b766-fwkvf: secret "image-registry-tls" not found
Jul 28 22:09:03.048680 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:03.048383    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edcb52a8-15ce-419d-93dc-cc405f75076e-registry-tls podName:edcb52a8-15ce-419d-93dc-cc405f75076e nodeName:}" failed. No retries permitted until 2026-07-28 22:09:05.04836488 +0000 UTC m=+242.428502517 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/edcb52a8-15ce-419d-93dc-cc405f75076e-registry-tls") pod "image-registry-7f9c64b766-fwkvf" (UID: "edcb52a8-15ce-419d-93dc-cc405f75076e") : secret "image-registry-tls" not found
Jul 28 22:09:05.065190 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:05.065144    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/edcb52a8-15ce-419d-93dc-cc405f75076e-registry-tls\") pod \"image-registry-7f9c64b766-fwkvf\" (UID: \"edcb52a8-15ce-419d-93dc-cc405f75076e\") " pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf"
Jul 28 22:09:05.065607 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:05.065300    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 22:09:05.065607 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:05.065327    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f9c64b766-fwkvf: secret "image-registry-tls" not found
Jul 28 22:09:05.065607 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:05.065401    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edcb52a8-15ce-419d-93dc-cc405f75076e-registry-tls podName:edcb52a8-15ce-419d-93dc-cc405f75076e nodeName:}" failed. No retries permitted until 2026-07-28 22:09:09.065384197 +0000 UTC m=+246.445521822 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/edcb52a8-15ce-419d-93dc-cc405f75076e-registry-tls") pod "image-registry-7f9c64b766-fwkvf" (UID: "edcb52a8-15ce-419d-93dc-cc405f75076e") : secret "image-registry-tls" not found
Jul 28 22:09:06.906126 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:06.906090    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bdmbq"]
Jul 28 22:09:06.908595 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:06.908567    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bdmbq"
Jul 28 22:09:06.911223 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:06.911203    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 22:09:06.911374 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:06.911239    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 22:09:06.911600 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:06.911579    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-t4cf9\""
Jul 28 22:09:06.911676 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:06.911582    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 22:09:06.911676 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:06.911640    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 22:09:06.921541 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:06.921515    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bdmbq"]
Jul 28 22:09:06.980809 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:06.980774    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f91a796a-dc22-4189-bb32-f78b985973c6-crio-socket\") pod \"insights-runtime-extractor-bdmbq\" (UID: \"f91a796a-dc22-4189-bb32-f78b985973c6\") " pod="openshift-insights/insights-runtime-extractor-bdmbq"
Jul 28 22:09:06.980966 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:06.980853    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chrmj\" (UniqueName: \"kubernetes.io/projected/f91a796a-dc22-4189-bb32-f78b985973c6-kube-api-access-chrmj\") pod \"insights-runtime-extractor-bdmbq\" (UID: \"f91a796a-dc22-4189-bb32-f78b985973c6\") " pod="openshift-insights/insights-runtime-extractor-bdmbq"
Jul 28 22:09:06.980966 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:06.980908    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f91a796a-dc22-4189-bb32-f78b985973c6-data-volume\") pod \"insights-runtime-extractor-bdmbq\" (UID: \"f91a796a-dc22-4189-bb32-f78b985973c6\") " pod="openshift-insights/insights-runtime-extractor-bdmbq"
Jul 28 22:09:06.980966 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:06.980927    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f91a796a-dc22-4189-bb32-f78b985973c6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bdmbq\" (UID: \"f91a796a-dc22-4189-bb32-f78b985973c6\") " pod="openshift-insights/insights-runtime-extractor-bdmbq"
Jul 28 22:09:06.980966 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:06.980943    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f91a796a-dc22-4189-bb32-f78b985973c6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdmbq\" (UID: \"f91a796a-dc22-4189-bb32-f78b985973c6\") " pod="openshift-insights/insights-runtime-extractor-bdmbq"
Jul 28 22:09:07.082223 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:07.082192    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chrmj\" (UniqueName: \"kubernetes.io/projected/f91a796a-dc22-4189-bb32-f78b985973c6-kube-api-access-chrmj\") pod \"insights-runtime-extractor-bdmbq\" (UID: \"f91a796a-dc22-4189-bb32-f78b985973c6\") " pod="openshift-insights/insights-runtime-extractor-bdmbq"
Jul 28 22:09:07.082387 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:07.082250    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f91a796a-dc22-4189-bb32-f78b985973c6-data-volume\") pod \"insights-runtime-extractor-bdmbq\" (UID: \"f91a796a-dc22-4189-bb32-f78b985973c6\") " pod="openshift-insights/insights-runtime-extractor-bdmbq"
Jul 28 22:09:07.082387 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:07.082274    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f91a796a-dc22-4189-bb32-f78b985973c6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bdmbq\" (UID: \"f91a796a-dc22-4189-bb32-f78b985973c6\") " pod="openshift-insights/insights-runtime-extractor-bdmbq"
Jul 28 22:09:07.082387 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:07.082301    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f91a796a-dc22-4189-bb32-f78b985973c6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdmbq\" (UID: \"f91a796a-dc22-4189-bb32-f78b985973c6\") " pod="openshift-insights/insights-runtime-extractor-bdmbq"
Jul 28 22:09:07.082387 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:07.082331    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f91a796a-dc22-4189-bb32-f78b985973c6-crio-socket\") pod \"insights-runtime-extractor-bdmbq\" (UID: \"f91a796a-dc22-4189-bb32-f78b985973c6\") " pod="openshift-insights/insights-runtime-extractor-bdmbq"
Jul 28 22:09:07.082523 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:07.082417    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 22:09:07.082523 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:07.082442    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f91a796a-dc22-4189-bb32-f78b985973c6-crio-socket\") pod \"insights-runtime-extractor-bdmbq\" (UID: \"f91a796a-dc22-4189-bb32-f78b985973c6\") " pod="openshift-insights/insights-runtime-extractor-bdmbq"
Jul 28 22:09:07.082523 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:07.082494    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f91a796a-dc22-4189-bb32-f78b985973c6-kube-rbac-proxy-cm podName:f91a796a-dc22-4189-bb32-f78b985973c6 nodeName:}" failed. No retries permitted until 2026-07-28 22:09:07.582476958 +0000 UTC m=+244.962614585 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f91a796a-dc22-4189-bb32-f78b985973c6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bdmbq" (UID: "f91a796a-dc22-4189-bb32-f78b985973c6") : configmap "kube-rbac-proxy" not found
Jul 28 22:09:07.082523 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:07.082514    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 22:09:07.082647 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:07.082581    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f91a796a-dc22-4189-bb32-f78b985973c6-insights-runtime-extractor-tls podName:f91a796a-dc22-4189-bb32-f78b985973c6 nodeName:}" failed. No retries permitted until 2026-07-28 22:09:07.58256498 +0000 UTC m=+244.962702603 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f91a796a-dc22-4189-bb32-f78b985973c6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bdmbq" (UID: "f91a796a-dc22-4189-bb32-f78b985973c6") : secret "insights-runtime-extractor-tls" not found
Jul 28 22:09:07.082647 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:07.082589    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f91a796a-dc22-4189-bb32-f78b985973c6-data-volume\") pod \"insights-runtime-extractor-bdmbq\" (UID: \"f91a796a-dc22-4189-bb32-f78b985973c6\") " pod="openshift-insights/insights-runtime-extractor-bdmbq"
Jul 28 22:09:07.101777 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:07.101743    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chrmj\" (UniqueName: \"kubernetes.io/projected/f91a796a-dc22-4189-bb32-f78b985973c6-kube-api-access-chrmj\") pod \"insights-runtime-extractor-bdmbq\" (UID: \"f91a796a-dc22-4189-bb32-f78b985973c6\") " pod="openshift-insights/insights-runtime-extractor-bdmbq"
Jul 28 22:09:07.586746 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:07.586708    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f91a796a-dc22-4189-bb32-f78b985973c6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bdmbq\" (UID: \"f91a796a-dc22-4189-bb32-f78b985973c6\") " pod="openshift-insights/insights-runtime-extractor-bdmbq"
Jul 28 22:09:07.586746 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:07.586747    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f91a796a-dc22-4189-bb32-f78b985973c6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdmbq\" (UID: \"f91a796a-dc22-4189-bb32-f78b985973c6\") " pod="openshift-insights/insights-runtime-extractor-bdmbq"
Jul 28 22:09:07.586969 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:07.586841    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 22:09:07.586969 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:07.586864    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 22:09:07.586969 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:07.586914    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f91a796a-dc22-4189-bb32-f78b985973c6-insights-runtime-extractor-tls podName:f91a796a-dc22-4189-bb32-f78b985973c6 nodeName:}" failed. No retries permitted until 2026-07-28 22:09:08.586901691 +0000 UTC m=+245.967039312 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f91a796a-dc22-4189-bb32-f78b985973c6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bdmbq" (UID: "f91a796a-dc22-4189-bb32-f78b985973c6") : secret "insights-runtime-extractor-tls" not found
Jul 28 22:09:07.586969 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:07.586929    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f91a796a-dc22-4189-bb32-f78b985973c6-kube-rbac-proxy-cm podName:f91a796a-dc22-4189-bb32-f78b985973c6 nodeName:}" failed. No retries permitted until 2026-07-28 22:09:08.586922832 +0000 UTC m=+245.967060454 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f91a796a-dc22-4189-bb32-f78b985973c6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bdmbq" (UID: "f91a796a-dc22-4189-bb32-f78b985973c6") : configmap "kube-rbac-proxy" not found
Jul 28 22:09:08.594320 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:08.594282    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f91a796a-dc22-4189-bb32-f78b985973c6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bdmbq\" (UID: \"f91a796a-dc22-4189-bb32-f78b985973c6\") " pod="openshift-insights/insights-runtime-extractor-bdmbq"
Jul 28 22:09:08.594320 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:08.594316    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f91a796a-dc22-4189-bb32-f78b985973c6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdmbq\" (UID: \"f91a796a-dc22-4189-bb32-f78b985973c6\") " pod="openshift-insights/insights-runtime-extractor-bdmbq"
Jul 28 22:09:08.594844 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:08.594412    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 22:09:08.594844 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:08.594426    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 22:09:08.594844 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:08.594474    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f91a796a-dc22-4189-bb32-f78b985973c6-insights-runtime-extractor-tls podName:f91a796a-dc22-4189-bb32-f78b985973c6 nodeName:}" failed. No retries permitted until 2026-07-28 22:09:10.594461074 +0000 UTC m=+247.974598696 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f91a796a-dc22-4189-bb32-f78b985973c6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bdmbq" (UID: "f91a796a-dc22-4189-bb32-f78b985973c6") : secret "insights-runtime-extractor-tls" not found
Jul 28 22:09:08.594844 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:08.594488    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f91a796a-dc22-4189-bb32-f78b985973c6-kube-rbac-proxy-cm podName:f91a796a-dc22-4189-bb32-f78b985973c6 nodeName:}" failed. No retries permitted until 2026-07-28 22:09:10.594481733 +0000 UTC m=+247.974619354 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f91a796a-dc22-4189-bb32-f78b985973c6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bdmbq" (UID: "f91a796a-dc22-4189-bb32-f78b985973c6") : configmap "kube-rbac-proxy" not found
Jul 28 22:09:09.097924 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:09.097889    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/edcb52a8-15ce-419d-93dc-cc405f75076e-registry-tls\") pod \"image-registry-7f9c64b766-fwkvf\" (UID: \"edcb52a8-15ce-419d-93dc-cc405f75076e\") " pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf"
Jul 28 22:09:09.098098 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:09.098000    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 22:09:09.098098 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:09.098011    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f9c64b766-fwkvf: secret "image-registry-tls" not found
Jul 28 22:09:09.098098 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:09.098058    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edcb52a8-15ce-419d-93dc-cc405f75076e-registry-tls podName:edcb52a8-15ce-419d-93dc-cc405f75076e nodeName:}" failed. No retries permitted until 2026-07-28 22:09:17.098044855 +0000 UTC m=+254.478182477 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/edcb52a8-15ce-419d-93dc-cc405f75076e-registry-tls") pod "image-registry-7f9c64b766-fwkvf" (UID: "edcb52a8-15ce-419d-93dc-cc405f75076e") : secret "image-registry-tls" not found
Jul 28 22:09:09.101658 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:09.101638    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qtgkr_820ee419-dab7-43ef-94bf-cadbaf9a5232/dns-node-resolver/0.log"
Jul 28 22:09:10.101271 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:10.101240    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tx5m6_2a9cac08-b862-4f7b-88e3-f4de98a035ea/node-ca/0.log"
Jul 28 22:09:10.610701 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:10.610631    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f91a796a-dc22-4189-bb32-f78b985973c6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bdmbq\" (UID: \"f91a796a-dc22-4189-bb32-f78b985973c6\") " pod="openshift-insights/insights-runtime-extractor-bdmbq"
Jul 28 22:09:10.610701 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:10.610703    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f91a796a-dc22-4189-bb32-f78b985973c6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdmbq\" (UID: \"f91a796a-dc22-4189-bb32-f78b985973c6\") " pod="openshift-insights/insights-runtime-extractor-bdmbq"
Jul 28 22:09:10.610945 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:10.610769    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 22:09:10.610945 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:10.610808    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 22:09:10.610945 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:10.610862    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f91a796a-dc22-4189-bb32-f78b985973c6-kube-rbac-proxy-cm podName:f91a796a-dc22-4189-bb32-f78b985973c6 nodeName:}" failed. No retries permitted until 2026-07-28 22:09:14.610846787 +0000 UTC m=+251.990984412 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f91a796a-dc22-4189-bb32-f78b985973c6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bdmbq" (UID: "f91a796a-dc22-4189-bb32-f78b985973c6") : configmap "kube-rbac-proxy" not found
Jul 28 22:09:10.610945 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:10.610877    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f91a796a-dc22-4189-bb32-f78b985973c6-insights-runtime-extractor-tls podName:f91a796a-dc22-4189-bb32-f78b985973c6 nodeName:}" failed. No retries permitted until 2026-07-28 22:09:14.610870711 +0000 UTC m=+251.991008332 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f91a796a-dc22-4189-bb32-f78b985973c6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bdmbq" (UID: "f91a796a-dc22-4189-bb32-f78b985973c6") : secret "insights-runtime-extractor-tls" not found
Jul 28 22:09:10.900316 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:10.900228    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ssgsl_d653889b-b9a3-4d5d-bbf0-f286413a20c9/serve-healthcheck-canary/0.log"
Jul 28 22:09:14.639281 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:14.639242    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f91a796a-dc22-4189-bb32-f78b985973c6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bdmbq\" (UID: \"f91a796a-dc22-4189-bb32-f78b985973c6\") " pod="openshift-insights/insights-runtime-extractor-bdmbq"
Jul 28 22:09:14.639281 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:14.639278    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f91a796a-dc22-4189-bb32-f78b985973c6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdmbq\" (UID: \"f91a796a-dc22-4189-bb32-f78b985973c6\") " pod="openshift-insights/insights-runtime-extractor-bdmbq"
Jul 28 22:09:14.639720 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:14.639397    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 22:09:14.639720 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:14.639411    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 22:09:14.639720 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:14.639463    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f91a796a-dc22-4189-bb32-f78b985973c6-kube-rbac-proxy-cm podName:f91a796a-dc22-4189-bb32-f78b985973c6 nodeName:}" failed. No retries permitted until 2026-07-28 22:09:22.63944673 +0000 UTC m=+260.019584353 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f91a796a-dc22-4189-bb32-f78b985973c6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bdmbq" (UID: "f91a796a-dc22-4189-bb32-f78b985973c6") : configmap "kube-rbac-proxy" not found
Jul 28 22:09:14.639720 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:14.639486    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f91a796a-dc22-4189-bb32-f78b985973c6-insights-runtime-extractor-tls podName:f91a796a-dc22-4189-bb32-f78b985973c6 nodeName:}" failed. No retries permitted until 2026-07-28 22:09:22.639476249 +0000 UTC m=+260.019613871 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f91a796a-dc22-4189-bb32-f78b985973c6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bdmbq" (UID: "f91a796a-dc22-4189-bb32-f78b985973c6") : secret "insights-runtime-extractor-tls" not found
Jul 28 22:09:14.941151 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:14.941124    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f33e531-3341-4c01-940e-91f17519976c-metrics-certs\") pod \"network-metrics-daemon-4xqhf\" (UID: \"2f33e531-3341-4c01-940e-91f17519976c\") " pod="openshift-multus/network-metrics-daemon-4xqhf"
Jul 28 22:09:14.941340 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:14.941289    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 28 22:09:14.941439 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:14.941395    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f33e531-3341-4c01-940e-91f17519976c-metrics-certs podName:2f33e531-3341-4c01-940e-91f17519976c nodeName:}" failed. No retries permitted until 2026-07-28 22:11:16.941370741 +0000 UTC m=+374.321508383 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f33e531-3341-4c01-940e-91f17519976c-metrics-certs") pod "network-metrics-daemon-4xqhf" (UID: "2f33e531-3341-4c01-940e-91f17519976c") : secret "metrics-daemon-secret" not found
Jul 28 22:09:17.158487 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:17.158446    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/edcb52a8-15ce-419d-93dc-cc405f75076e-registry-tls\") pod \"image-registry-7f9c64b766-fwkvf\" (UID: \"edcb52a8-15ce-419d-93dc-cc405f75076e\") " pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf"
Jul 28 22:09:17.160807 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:17.160784    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/edcb52a8-15ce-419d-93dc-cc405f75076e-registry-tls\") pod \"image-registry-7f9c64b766-fwkvf\" (UID: \"edcb52a8-15ce-419d-93dc-cc405f75076e\") " pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf"
Jul 28 22:09:17.217550 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:17.217521    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8lc2p\""
Jul 28 22:09:17.225793 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:17.225772    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf"
Jul 28 22:09:17.362679 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:17.362650    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f9c64b766-fwkvf"]
Jul 28 22:09:17.366583 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:09:17.366554    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedcb52a8_15ce_419d_93dc_cc405f75076e.slice/crio-c9cc94d946a9476810a1430e4872c581f78335d8cf08af0b5f6b4ee0bec91979 WatchSource:0}: Error finding container c9cc94d946a9476810a1430e4872c581f78335d8cf08af0b5f6b4ee0bec91979: Status 404 returned error can't find the container with id c9cc94d946a9476810a1430e4872c581f78335d8cf08af0b5f6b4ee0bec91979
Jul 28 22:09:17.892478 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:17.892444    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf" event={"ID":"edcb52a8-15ce-419d-93dc-cc405f75076e","Type":"ContainerStarted","Data":"172601c8d016666801d9f5aa3e7d600e84aba90fdc641cd9a5dba90f6f4c00ca"}
Jul 28 22:09:17.892654 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:17.892492    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf" event={"ID":"edcb52a8-15ce-419d-93dc-cc405f75076e","Type":"ContainerStarted","Data":"c9cc94d946a9476810a1430e4872c581f78335d8cf08af0b5f6b4ee0bec91979"}
Jul 28 22:09:17.892654 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:17.892519    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf"
Jul 28 22:09:17.916233 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:17.916191    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf" podStartSLOduration=16.916176692 podStartE2EDuration="16.916176692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:09:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:09:17.915317909 +0000 UTC m=+255.295455554" watchObservedRunningTime="2026-07-28 22:09:17.916176692 +0000 UTC m=+255.296314335"
Jul 28 22:09:22.699653 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:22.699607    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f91a796a-dc22-4189-bb32-f78b985973c6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bdmbq\" (UID: \"f91a796a-dc22-4189-bb32-f78b985973c6\") " pod="openshift-insights/insights-runtime-extractor-bdmbq"
Jul 28 22:09:22.699653 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:22.699652    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f91a796a-dc22-4189-bb32-f78b985973c6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdmbq\" (UID: \"f91a796a-dc22-4189-bb32-f78b985973c6\") " pod="openshift-insights/insights-runtime-extractor-bdmbq"
Jul 28 22:09:22.700122 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:22.700102    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f91a796a-dc22-4189-bb32-f78b985973c6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bdmbq\" (UID: \"f91a796a-dc22-4189-bb32-f78b985973c6\") " pod="openshift-insights/insights-runtime-extractor-bdmbq"
Jul 28 22:09:22.701934 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:22.701913    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f91a796a-dc22-4189-bb32-f78b985973c6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdmbq\" (UID: \"f91a796a-dc22-4189-bb32-f78b985973c6\") " pod="openshift-insights/insights-runtime-extractor-bdmbq"
Jul 28 22:09:22.817366 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:22.817320    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bdmbq"
Jul 28 22:09:22.939996 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:22.939952    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bdmbq"]
Jul 28 22:09:22.942758 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:09:22.942730    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf91a796a_dc22_4189_bb32_f78b985973c6.slice/crio-f78f3e59f5e1a8d8290af7f88e5266959d83312911c13868f4955c4759d90583 WatchSource:0}: Error finding container f78f3e59f5e1a8d8290af7f88e5266959d83312911c13868f4955c4759d90583: Status 404 returned error can't find the container with id f78f3e59f5e1a8d8290af7f88e5266959d83312911c13868f4955c4759d90583
Jul 28 22:09:23.913195 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:23.913122    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bdmbq" event={"ID":"f91a796a-dc22-4189-bb32-f78b985973c6","Type":"ContainerStarted","Data":"f78f3e59f5e1a8d8290af7f88e5266959d83312911c13868f4955c4759d90583"}
Jul 28 22:09:27.093079 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:27.092992    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f9c64b766-fwkvf"]
Jul 28 22:09:27.150183 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:27.150154    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-86664d9944-5hsqd"]
Jul 28 22:09:27.153104 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:27.153085    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86664d9944-5hsqd"
Jul 28 22:09:27.169866 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:27.169828    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86664d9944-5hsqd"]
Jul 28 22:09:27.235511 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:27.235480    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bb491a55-2f3a-4096-a4b1-e09b62d3bef9-registry-certificates\") pod \"image-registry-86664d9944-5hsqd\" (UID: \"bb491a55-2f3a-4096-a4b1-e09b62d3bef9\") " pod="openshift-image-registry/image-registry-86664d9944-5hsqd"
Jul 28 22:09:27.235699 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:27.235527    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t67bg\" (UniqueName: \"kubernetes.io/projected/bb491a55-2f3a-4096-a4b1-e09b62d3bef9-kube-api-access-t67bg\") pod \"image-registry-86664d9944-5hsqd\" (UID: \"bb491a55-2f3a-4096-a4b1-e09b62d3bef9\") " pod="openshift-image-registry/image-registry-86664d9944-5hsqd"
Jul 28 22:09:27.235699 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:27.235581    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bb491a55-2f3a-4096-a4b1-e09b62d3bef9-trusted-ca\") pod \"image-registry-86664d9944-5hsqd\" (UID: \"bb491a55-2f3a-4096-a4b1-e09b62d3bef9\") " pod="openshift-image-registry/image-registry-86664d9944-5hsqd"
Jul 28 22:09:27.235699 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:27.235636    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bb491a55-2f3a-4096-a4b1-e09b62d3bef9-ca-trust-extracted\") pod \"image-registry-86664d9944-5hsqd\" (UID: \"bb491a55-2f3a-4096-a4b1-e09b62d3bef9\") " pod="openshift-image-registry/image-registry-86664d9944-5hsqd"
Jul 28 22:09:27.235699 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:27.235658    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bb491a55-2f3a-4096-a4b1-e09b62d3bef9-bound-sa-token\") pod \"image-registry-86664d9944-5hsqd\" (UID: \"bb491a55-2f3a-4096-a4b1-e09b62d3bef9\") " pod="openshift-image-registry/image-registry-86664d9944-5hsqd"
Jul 28 22:09:27.235699 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:27.235683    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bb491a55-2f3a-4096-a4b1-e09b62d3bef9-image-registry-private-configuration\") pod \"image-registry-86664d9944-5hsqd\" (UID: \"bb491a55-2f3a-4096-a4b1-e09b62d3bef9\") " pod="openshift-image-registry/image-registry-86664d9944-5hsqd"
Jul 28 22:09:27.235915 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:27.235754    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bb491a55-2f3a-4096-a4b1-e09b62d3bef9-installation-pull-secrets\") pod \"image-registry-86664d9944-5hsqd\" (UID: \"bb491a55-2f3a-4096-a4b1-e09b62d3bef9\") " pod="openshift-image-registry/image-registry-86664d9944-5hsqd"
Jul 28 22:09:27.235915 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:27.235824    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bb491a55-2f3a-4096-a4b1-e09b62d3bef9-registry-tls\") pod \"image-registry-86664d9944-5hsqd\" (UID: \"bb491a55-2f3a-4096-a4b1-e09b62d3bef9\") " pod="openshift-image-registry/image-registry-86664d9944-5hsqd"
Jul 28 22:09:27.336476 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:27.336444    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bb491a55-2f3a-4096-a4b1-e09b62d3bef9-registry-tls\") pod \"image-registry-86664d9944-5hsqd\" (UID: \"bb491a55-2f3a-4096-a4b1-e09b62d3bef9\") " pod="openshift-image-registry/image-registry-86664d9944-5hsqd"
Jul 28 22:09:27.336655 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:27.336486    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bb491a55-2f3a-4096-a4b1-e09b62d3bef9-registry-certificates\") pod \"image-registry-86664d9944-5hsqd\" (UID: \"bb491a55-2f3a-4096-a4b1-e09b62d3bef9\") " pod="openshift-image-registry/image-registry-86664d9944-5hsqd"
Jul 28 22:09:27.336655 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:27.336506    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t67bg\" (UniqueName: \"kubernetes.io/projected/bb491a55-2f3a-4096-a4b1-e09b62d3bef9-kube-api-access-t67bg\") pod \"image-registry-86664d9944-5hsqd\" (UID: \"bb491a55-2f3a-4096-a4b1-e09b62d3bef9\") " pod="openshift-image-registry/image-registry-86664d9944-5hsqd"
Jul 28 22:09:27.336655 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:27.336526    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bb491a55-2f3a-4096-a4b1-e09b62d3bef9-trusted-ca\") pod \"image-registry-86664d9944-5hsqd\" (UID: \"bb491a55-2f3a-4096-a4b1-e09b62d3bef9\") " pod="openshift-image-registry/image-registry-86664d9944-5hsqd"
Jul 28 22:09:27.336655 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:27.336543    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bb491a55-2f3a-4096-a4b1-e09b62d3bef9-ca-trust-extracted\") pod \"image-registry-86664d9944-5hsqd\" (UID: \"bb491a55-2f3a-4096-a4b1-e09b62d3bef9\") " pod="openshift-image-registry/image-registry-86664d9944-5hsqd"
Jul 28 22:09:27.336655 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:27.336557    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bb491a55-2f3a-4096-a4b1-e09b62d3bef9-bound-sa-token\") pod \"image-registry-86664d9944-5hsqd\" (UID: \"bb491a55-2f3a-4096-a4b1-e09b62d3bef9\") " pod="openshift-image-registry/image-registry-86664d9944-5hsqd"
Jul 28 22:09:27.336655 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:27.336575    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bb491a55-2f3a-4096-a4b1-e09b62d3bef9-image-registry-private-configuration\") pod \"image-registry-86664d9944-5hsqd\" (UID: \"bb491a55-2f3a-4096-a4b1-e09b62d3bef9\") " pod="openshift-image-registry/image-registry-86664d9944-5hsqd"
Jul 28 22:09:27.336655 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:27.336625    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bb491a55-2f3a-4096-a4b1-e09b62d3bef9-installation-pull-secrets\") pod \"image-registry-86664d9944-5hsqd\" (UID: \"bb491a55-2f3a-4096-a4b1-e09b62d3bef9\") " pod="openshift-image-registry/image-registry-86664d9944-5hsqd"
Jul 28 22:09:27.337391 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:27.337309    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bb491a55-2f3a-4096-a4b1-e09b62d3bef9-ca-trust-extracted\") pod \"image-registry-86664d9944-5hsqd\" (UID: \"bb491a55-2f3a-4096-a4b1-e09b62d3bef9\") " pod="openshift-image-registry/image-registry-86664d9944-5hsqd"
Jul 28 22:09:27.337514 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:27.337424    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bb491a55-2f3a-4096-a4b1-e09b62d3bef9-registry-certificates\") pod \"image-registry-86664d9944-5hsqd\" (UID: \"bb491a55-2f3a-4096-a4b1-e09b62d3bef9\") " pod="openshift-image-registry/image-registry-86664d9944-5hsqd"
Jul 28 22:09:27.337572 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:27.337545    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bb491a55-2f3a-4096-a4b1-e09b62d3bef9-trusted-ca\") pod \"image-registry-86664d9944-5hsqd\" (UID: \"bb491a55-2f3a-4096-a4b1-e09b62d3bef9\") " pod="openshift-image-registry/image-registry-86664d9944-5hsqd"
Jul 28 22:09:27.339549 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:27.339520    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bb491a55-2f3a-4096-a4b1-e09b62d3bef9-installation-pull-secrets\") pod \"image-registry-86664d9944-5hsqd\" (UID: \"bb491a55-2f3a-4096-a4b1-e09b62d3bef9\") " pod="openshift-image-registry/image-registry-86664d9944-5hsqd"
Jul 28 22:09:27.339680 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:27.339657    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bb491a55-2f3a-4096-a4b1-e09b62d3bef9-image-registry-private-configuration\") pod \"image-registry-86664d9944-5hsqd\" (UID: \"bb491a55-2f3a-4096-a4b1-e09b62d3bef9\") " pod="openshift-image-registry/image-registry-86664d9944-5hsqd"
Jul 28 22:09:27.339787 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:27.339770    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bb491a55-2f3a-4096-a4b1-e09b62d3bef9-registry-tls\") pod \"image-registry-86664d9944-5hsqd\" (UID: \"bb491a55-2f3a-4096-a4b1-e09b62d3bef9\") " pod="openshift-image-registry/image-registry-86664d9944-5hsqd"
Jul 28 22:09:27.344481 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:27.344399    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bb491a55-2f3a-4096-a4b1-e09b62d3bef9-bound-sa-token\") pod \"image-registry-86664d9944-5hsqd\" (UID: \"bb491a55-2f3a-4096-a4b1-e09b62d3bef9\") " pod="openshift-image-registry/image-registry-86664d9944-5hsqd"
Jul 28 22:09:27.344644 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:27.344628    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t67bg\" (UniqueName: \"kubernetes.io/projected/bb491a55-2f3a-4096-a4b1-e09b62d3bef9-kube-api-access-t67bg\") pod \"image-registry-86664d9944-5hsqd\" (UID: \"bb491a55-2f3a-4096-a4b1-e09b62d3bef9\") " pod="openshift-image-registry/image-registry-86664d9944-5hsqd"
Jul 28 22:09:27.461603 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:27.461569    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86664d9944-5hsqd"
Jul 28 22:09:27.914934 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:27.914909    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86664d9944-5hsqd"]
Jul 28 22:09:27.917329 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:09:27.917297    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb491a55_2f3a_4096_a4b1_e09b62d3bef9.slice/crio-f59319cc7b1ee858978a32b8d3f71ada65ff16c94e9baab1f02e9c83ebc31ee3 WatchSource:0}: Error finding container f59319cc7b1ee858978a32b8d3f71ada65ff16c94e9baab1f02e9c83ebc31ee3: Status 404 returned error can't find the container with id f59319cc7b1ee858978a32b8d3f71ada65ff16c94e9baab1f02e9c83ebc31ee3
Jul 28 22:09:27.934490 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:27.934386    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bdmbq" event={"ID":"f91a796a-dc22-4189-bb32-f78b985973c6","Type":"ContainerStarted","Data":"59eee7146e09d3349b68f1368a5298c8cf0f7acb285bf7abf100ad55ce5281d9"}
Jul 28 22:09:27.935725 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:27.935704    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86664d9944-5hsqd" event={"ID":"bb491a55-2f3a-4096-a4b1-e09b62d3bef9","Type":"ContainerStarted","Data":"f59319cc7b1ee858978a32b8d3f71ada65ff16c94e9baab1f02e9c83ebc31ee3"}
Jul 28 22:09:28.940791 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:28.940753    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86664d9944-5hsqd" event={"ID":"bb491a55-2f3a-4096-a4b1-e09b62d3bef9","Type":"ContainerStarted","Data":"9dff0fa78b8560b1f448f49265c9b4c76fb4bcb444751af2cccb968fd8b118c2"}
Jul 28 22:09:28.941260 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:28.940905    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-86664d9944-5hsqd"
Jul 28 22:09:31.952632 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:31.952598    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bdmbq" event={"ID":"f91a796a-dc22-4189-bb32-f78b985973c6","Type":"ContainerStarted","Data":"36b2522f66ae31aba8e27eac79e4b64702ecc7c63a8fcc88fd4f633d9f941b8c"}
Jul 28 22:09:34.963887 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:34.963853    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bdmbq" event={"ID":"f91a796a-dc22-4189-bb32-f78b985973c6","Type":"ContainerStarted","Data":"890e43df6dabb171b0ffc8c06c7451a7b5f10335123e8988aff778adfe7db1a9"}
Jul 28 22:09:35.027002 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:35.026949    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bdmbq" podStartSLOduration=17.473977485 podStartE2EDuration="29.02693443s" totalImagesPullingTime="11.552956945s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:09:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:09:35.025470673 +0000 UTC m=+272.405608318" watchObservedRunningTime="2026-07-28 22:09:35.02693443 +0000 UTC m=+272.407072074"
Jul 28 22:09:35.027169 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:35.027046    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-86664d9944-5hsqd" podStartSLOduration=8.027040795 podStartE2EDuration="8.027040795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:09:28.959915608 +0000 UTC m=+266.340053253" watchObservedRunningTime="2026-07-28 22:09:35.027040795 +0000 UTC m=+272.407178439"
Jul 28 22:09:37.098520 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:37.098486    2563 patch_prober.go:28] interesting pod/image-registry-7f9c64b766-fwkvf container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 28 22:09:37.098941 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:37.098545    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf" podUID="edcb52a8-15ce-419d-93dc-cc405f75076e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 22:09:37.226054 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:37.226022    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-lwvw5"]
Jul 28 22:09:37.229080 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:37.229064    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-lwvw5"
Jul 28 22:09:37.231894 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:37.231868    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 28 22:09:37.232013 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:37.231895    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-d7hr2\""
Jul 28 22:09:37.239090 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:37.239067    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-lwvw5"]
Jul 28 22:09:37.316941 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:37.316907    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/85f15809-aac0-4ccb-a862-7d1ea31d353b-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-lwvw5\" (UID: \"85f15809-aac0-4ccb-a862-7d1ea31d353b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-lwvw5"
Jul 28 22:09:37.417395 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:37.417303    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/85f15809-aac0-4ccb-a862-7d1ea31d353b-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-lwvw5\" (UID: \"85f15809-aac0-4ccb-a862-7d1ea31d353b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-lwvw5"
Jul 28 22:09:37.419800 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:37.419776    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/85f15809-aac0-4ccb-a862-7d1ea31d353b-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-lwvw5\" (UID: \"85f15809-aac0-4ccb-a862-7d1ea31d353b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-lwvw5"
Jul 28 22:09:37.538159 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:37.538126    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-lwvw5"
Jul 28 22:09:37.660436 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:37.660405    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-lwvw5"]
Jul 28 22:09:37.663102 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:09:37.663073    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f15809_aac0_4ccb_a862_7d1ea31d353b.slice/crio-bcf18a0e3ef5716efa72ee525c51fa89e36c3d19a382729e5f8cc4b731989d8b WatchSource:0}: Error finding container bcf18a0e3ef5716efa72ee525c51fa89e36c3d19a382729e5f8cc4b731989d8b: Status 404 returned error can't find the container with id bcf18a0e3ef5716efa72ee525c51fa89e36c3d19a382729e5f8cc4b731989d8b
Jul 28 22:09:37.975699 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:37.975664    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-lwvw5" event={"ID":"85f15809-aac0-4ccb-a862-7d1ea31d353b","Type":"ContainerStarted","Data":"bcf18a0e3ef5716efa72ee525c51fa89e36c3d19a382729e5f8cc4b731989d8b"}
Jul 28 22:09:38.984192 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:38.984096    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-lwvw5" event={"ID":"85f15809-aac0-4ccb-a862-7d1ea31d353b","Type":"ContainerStarted","Data":"f75bd927888fc5a3c7b32d645534ef1c7a22f44efa0d58279c460469c6ee4615"}
Jul 28 22:09:38.984644 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:38.984317    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-lwvw5"
Jul 28 22:09:38.988852 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:38.988830    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-lwvw5"
Jul 28 22:09:39.007413 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:39.007371    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-lwvw5" podStartSLOduration=0.974211534 podStartE2EDuration="2.007341647s" totalImagesPullingTime="1.033130113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:09:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:09:39.007156915 +0000 UTC m=+276.387294572" watchObservedRunningTime="2026-07-28 22:09:39.007341647 +0000 UTC m=+276.387479290"
Jul 28 22:09:39.274687 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:39.274606    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-fcjgp"]
Jul 28 22:09:39.277629 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:39.277614    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-fcjgp"
Jul 28 22:09:39.280308 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:39.280285    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 22:09:39.280412 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:39.280372    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 22:09:39.280484 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:39.280412    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 22:09:39.280484 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:39.280437    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 28 22:09:39.280484 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:39.280447    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 28 22:09:39.280750 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:39.280734    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-w6w2c\""
Jul 28 22:09:39.286322 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:39.286303    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-fcjgp"]
Jul 28 22:09:39.333454 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:39.333417    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3d007a2c-32b5-4539-9b07-4b4a27973482-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-fcjgp\" (UID: \"3d007a2c-32b5-4539-9b07-4b4a27973482\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-fcjgp"
Jul 28 22:09:39.333454 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:39.333459    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8grw\" (UniqueName: \"kubernetes.io/projected/3d007a2c-32b5-4539-9b07-4b4a27973482-kube-api-access-j8grw\") pod \"prometheus-operator-84d69f794d-fcjgp\" (UID: \"3d007a2c-32b5-4539-9b07-4b4a27973482\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-fcjgp"
Jul 28 22:09:39.333633 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:39.333488    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d007a2c-32b5-4539-9b07-4b4a27973482-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-fcjgp\" (UID: \"3d007a2c-32b5-4539-9b07-4b4a27973482\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-fcjgp"
Jul 28 22:09:39.333633 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:39.333507    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3d007a2c-32b5-4539-9b07-4b4a27973482-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-fcjgp\" (UID: \"3d007a2c-32b5-4539-9b07-4b4a27973482\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-fcjgp"
Jul 28 22:09:39.433817 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:39.433784    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3d007a2c-32b5-4539-9b07-4b4a27973482-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-fcjgp\" (UID: \"3d007a2c-32b5-4539-9b07-4b4a27973482\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-fcjgp"
Jul 28 22:09:39.433817 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:39.433820    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8grw\" (UniqueName: \"kubernetes.io/projected/3d007a2c-32b5-4539-9b07-4b4a27973482-kube-api-access-j8grw\") pod \"prometheus-operator-84d69f794d-fcjgp\" (UID: \"3d007a2c-32b5-4539-9b07-4b4a27973482\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-fcjgp"
Jul 28 22:09:39.433966 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:39.433844    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d007a2c-32b5-4539-9b07-4b4a27973482-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-fcjgp\" (UID: \"3d007a2c-32b5-4539-9b07-4b4a27973482\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-fcjgp"
Jul 28 22:09:39.433966 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:39.433865    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3d007a2c-32b5-4539-9b07-4b4a27973482-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-fcjgp\" (UID: \"3d007a2c-32b5-4539-9b07-4b4a27973482\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-fcjgp"
Jul 28 22:09:39.433966 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:39.433932    2563 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 28 22:09:39.434059 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:39.433991    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d007a2c-32b5-4539-9b07-4b4a27973482-prometheus-operator-tls podName:3d007a2c-32b5-4539-9b07-4b4a27973482 nodeName:}" failed. No retries permitted until 2026-07-28 22:09:39.933975422 +0000 UTC m=+277.314113047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/3d007a2c-32b5-4539-9b07-4b4a27973482-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-fcjgp" (UID: "3d007a2c-32b5-4539-9b07-4b4a27973482") : secret "prometheus-operator-tls" not found
Jul 28 22:09:39.434612 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:39.434592    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d007a2c-32b5-4539-9b07-4b4a27973482-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-fcjgp\" (UID: \"3d007a2c-32b5-4539-9b07-4b4a27973482\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-fcjgp"
Jul 28 22:09:39.436204 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:39.436183    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3d007a2c-32b5-4539-9b07-4b4a27973482-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-fcjgp\" (UID: \"3d007a2c-32b5-4539-9b07-4b4a27973482\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-fcjgp"
Jul 28 22:09:39.442583 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:39.442560    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8grw\" (UniqueName: \"kubernetes.io/projected/3d007a2c-32b5-4539-9b07-4b4a27973482-kube-api-access-j8grw\") pod \"prometheus-operator-84d69f794d-fcjgp\" (UID: \"3d007a2c-32b5-4539-9b07-4b4a27973482\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-fcjgp"
Jul 28 22:09:39.936096 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:39.936062    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3d007a2c-32b5-4539-9b07-4b4a27973482-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-fcjgp\" (UID: \"3d007a2c-32b5-4539-9b07-4b4a27973482\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-fcjgp"
Jul 28 22:09:39.938362 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:39.938321    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3d007a2c-32b5-4539-9b07-4b4a27973482-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-fcjgp\" (UID: \"3d007a2c-32b5-4539-9b07-4b4a27973482\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-fcjgp"
Jul 28 22:09:40.187569 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:40.187490    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-fcjgp"
Jul 28 22:09:40.313094 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:40.313061    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-fcjgp"]
Jul 28 22:09:40.315866 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:09:40.315837    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d007a2c_32b5_4539_9b07_4b4a27973482.slice/crio-5a733cb86e0a9a58cfd592abdadd54c0c46fdbe73610ee350a267e2decbda27e WatchSource:0}: Error finding container 5a733cb86e0a9a58cfd592abdadd54c0c46fdbe73610ee350a267e2decbda27e: Status 404 returned error can't find the container with id 5a733cb86e0a9a58cfd592abdadd54c0c46fdbe73610ee350a267e2decbda27e
Jul 28 22:09:40.991154 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:40.991112    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-fcjgp" event={"ID":"3d007a2c-32b5-4539-9b07-4b4a27973482","Type":"ContainerStarted","Data":"5a733cb86e0a9a58cfd592abdadd54c0c46fdbe73610ee350a267e2decbda27e"}
Jul 28 22:09:41.997803 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:41.997700    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-fcjgp" event={"ID":"3d007a2c-32b5-4539-9b07-4b4a27973482","Type":"ContainerStarted","Data":"5893d8199b004f172f6fd4a8758cfcb44db9e852b2c29281867a59f58fb0096e"}
Jul 28 22:09:41.997803 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:41.997773    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-fcjgp" event={"ID":"3d007a2c-32b5-4539-9b07-4b4a27973482","Type":"ContainerStarted","Data":"99ee719a9494cdbaca52c079f322e08425cf5394906a95abe00c0850dd2a0252"}
Jul 28 22:09:42.018335 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:42.018287    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-fcjgp" podStartSLOduration=1.595109656 podStartE2EDuration="3.018268363s" totalImagesPullingTime="1.423158707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:09:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:09:42.017392379 +0000 UTC m=+279.397530024" watchObservedRunningTime="2026-07-28 22:09:42.018268363 +0000 UTC m=+279.398406008"
Jul 28 22:09:43.656938 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:43.656906    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dpmd2"]
Jul 28 22:09:43.660192 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:43.660171    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dpmd2"
Jul 28 22:09:43.662784 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:43.662763    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 22:09:43.662873 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:43.662798    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 22:09:43.662873 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:43.662808    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 22:09:43.663049 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:43.663032    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rhrfd\""
Jul 28 22:09:43.765308 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:43.765283    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59091947-e558-4909-8c6a-85a008823b1c-sys\") pod \"node-exporter-dpmd2\" (UID: \"59091947-e558-4909-8c6a-85a008823b1c\") " pod="openshift-monitoring/node-exporter-dpmd2"
Jul 28 22:09:43.765308 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:43.765316    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/59091947-e558-4909-8c6a-85a008823b1c-node-exporter-wtmp\") pod \"node-exporter-dpmd2\" (UID: \"59091947-e558-4909-8c6a-85a008823b1c\") " pod="openshift-monitoring/node-exporter-dpmd2"
Jul 28 22:09:43.765540 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:43.765341    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59091947-e558-4909-8c6a-85a008823b1c-metrics-client-ca\") pod \"node-exporter-dpmd2\" (UID: \"59091947-e558-4909-8c6a-85a008823b1c\") " pod="openshift-monitoring/node-exporter-dpmd2"
Jul 28 22:09:43.765540 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:43.765418    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/59091947-e558-4909-8c6a-85a008823b1c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dpmd2\" (UID: \"59091947-e558-4909-8c6a-85a008823b1c\") " pod="openshift-monitoring/node-exporter-dpmd2"
Jul 28 22:09:43.765540 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:43.765445    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/59091947-e558-4909-8c6a-85a008823b1c-node-exporter-tls\") pod \"node-exporter-dpmd2\" (UID: \"59091947-e558-4909-8c6a-85a008823b1c\") " pod="openshift-monitoring/node-exporter-dpmd2"
Jul 28 22:09:43.765540 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:43.765461    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2svj\" (UniqueName: \"kubernetes.io/projected/59091947-e558-4909-8c6a-85a008823b1c-kube-api-access-j2svj\") pod \"node-exporter-dpmd2\" (UID: \"59091947-e558-4909-8c6a-85a008823b1c\") " pod="openshift-monitoring/node-exporter-dpmd2"
Jul 28 22:09:43.765540 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:43.765532    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/59091947-e558-4909-8c6a-85a008823b1c-node-exporter-accelerators-collector-config\") pod \"node-exporter-dpmd2\" (UID: \"59091947-e558-4909-8c6a-85a008823b1c\") " pod="openshift-monitoring/node-exporter-dpmd2"
Jul 28 22:09:43.765760 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:43.765584    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/59091947-e558-4909-8c6a-85a008823b1c-node-exporter-textfile\") pod \"node-exporter-dpmd2\" (UID: \"59091947-e558-4909-8c6a-85a008823b1c\") " pod="openshift-monitoring/node-exporter-dpmd2"
Jul 28 22:09:43.765760 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:43.765615    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/59091947-e558-4909-8c6a-85a008823b1c-root\") pod \"node-exporter-dpmd2\" (UID: \"59091947-e558-4909-8c6a-85a008823b1c\") " pod="openshift-monitoring/node-exporter-dpmd2"
Jul 28 22:09:43.866750 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:43.866719    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59091947-e558-4909-8c6a-85a008823b1c-sys\") pod \"node-exporter-dpmd2\" (UID: \"59091947-e558-4909-8c6a-85a008823b1c\") " pod="openshift-monitoring/node-exporter-dpmd2"
Jul 28 22:09:43.866913 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:43.866758    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/59091947-e558-4909-8c6a-85a008823b1c-node-exporter-wtmp\") pod \"node-exporter-dpmd2\" (UID: \"59091947-e558-4909-8c6a-85a008823b1c\") " pod="openshift-monitoring/node-exporter-dpmd2"
Jul 28 22:09:43.866913 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:43.866781    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59091947-e558-4909-8c6a-85a008823b1c-metrics-client-ca\") pod \"node-exporter-dpmd2\" (UID: \"59091947-e558-4909-8c6a-85a008823b1c\") " pod="openshift-monitoring/node-exporter-dpmd2"
Jul 28 22:09:43.866913 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:43.866835    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59091947-e558-4909-8c6a-85a008823b1c-sys\") pod \"node-exporter-dpmd2\" (UID: \"59091947-e558-4909-8c6a-85a008823b1c\") " pod="openshift-monitoring/node-exporter-dpmd2"
Jul 28 22:09:43.866913 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:43.866899    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/59091947-e558-4909-8c6a-85a008823b1c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dpmd2\" (UID: \"59091947-e558-4909-8c6a-85a008823b1c\") " pod="openshift-monitoring/node-exporter-dpmd2"
Jul 28 22:09:43.867043 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:43.866913    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/59091947-e558-4909-8c6a-85a008823b1c-node-exporter-wtmp\") pod \"node-exporter-dpmd2\" (UID: \"59091947-e558-4909-8c6a-85a008823b1c\") " pod="openshift-monitoring/node-exporter-dpmd2"
Jul 28 22:09:43.867043 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:43.866925    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/59091947-e558-4909-8c6a-85a008823b1c-node-exporter-tls\") pod \"node-exporter-dpmd2\" (UID: \"59091947-e558-4909-8c6a-85a008823b1c\") " pod="openshift-monitoring/node-exporter-dpmd2"
Jul 28 22:09:43.867043 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:43.866960    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2svj\" (UniqueName: \"kubernetes.io/projected/59091947-e558-4909-8c6a-85a008823b1c-kube-api-access-j2svj\") pod \"node-exporter-dpmd2\" (UID: \"59091947-e558-4909-8c6a-85a008823b1c\") " pod="openshift-monitoring/node-exporter-dpmd2"
Jul 28 22:09:43.867043 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:43.866999    2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 28 22:09:43.867043 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:43.867004    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/59091947-e558-4909-8c6a-85a008823b1c-node-exporter-accelerators-collector-config\") pod \"node-exporter-dpmd2\" (UID: \"59091947-e558-4909-8c6a-85a008823b1c\") " pod="openshift-monitoring/node-exporter-dpmd2"
Jul 28 22:09:43.867043 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:43.867043    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59091947-e558-4909-8c6a-85a008823b1c-node-exporter-tls podName:59091947-e558-4909-8c6a-85a008823b1c nodeName:}" failed. No retries permitted until 2026-07-28 22:09:44.367027203 +0000 UTC m=+281.747164825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/59091947-e558-4909-8c6a-85a008823b1c-node-exporter-tls") pod "node-exporter-dpmd2" (UID: "59091947-e558-4909-8c6a-85a008823b1c") : secret "node-exporter-tls" not found
Jul 28 22:09:43.867306 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:43.867072    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/59091947-e558-4909-8c6a-85a008823b1c-node-exporter-textfile\") pod \"node-exporter-dpmd2\" (UID: \"59091947-e558-4909-8c6a-85a008823b1c\") " pod="openshift-monitoring/node-exporter-dpmd2"
Jul 28 22:09:43.867306 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:43.867103    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/59091947-e558-4909-8c6a-85a008823b1c-root\") pod \"node-exporter-dpmd2\" (UID: \"59091947-e558-4909-8c6a-85a008823b1c\") " pod="openshift-monitoring/node-exporter-dpmd2"
Jul 28 22:09:43.867306 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:43.867193    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/59091947-e558-4909-8c6a-85a008823b1c-root\") pod \"node-exporter-dpmd2\" (UID: \"59091947-e558-4909-8c6a-85a008823b1c\") " pod="openshift-monitoring/node-exporter-dpmd2"
Jul 28 22:09:43.867477 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:43.867382    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59091947-e558-4909-8c6a-85a008823b1c-metrics-client-ca\") pod \"node-exporter-dpmd2\" (UID: \"59091947-e558-4909-8c6a-85a008823b1c\") " pod="openshift-monitoring/node-exporter-dpmd2"
Jul 28 22:09:43.867477 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:43.867409    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/59091947-e558-4909-8c6a-85a008823b1c-node-exporter-textfile\") pod \"node-exporter-dpmd2\" (UID: \"59091947-e558-4909-8c6a-85a008823b1c\") " pod="openshift-monitoring/node-exporter-dpmd2"
Jul 28 22:09:43.867542 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:43.867494    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/59091947-e558-4909-8c6a-85a008823b1c-node-exporter-accelerators-collector-config\") pod \"node-exporter-dpmd2\" (UID: \"59091947-e558-4909-8c6a-85a008823b1c\") " pod="openshift-monitoring/node-exporter-dpmd2"
Jul 28 22:09:43.869164 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:43.869144    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/59091947-e558-4909-8c6a-85a008823b1c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dpmd2\" (UID: \"59091947-e558-4909-8c6a-85a008823b1c\") " pod="openshift-monitoring/node-exporter-dpmd2"
Jul 28 22:09:43.876396 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:43.876343    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2svj\" (UniqueName: \"kubernetes.io/projected/59091947-e558-4909-8c6a-85a008823b1c-kube-api-access-j2svj\") pod \"node-exporter-dpmd2\" (UID: \"59091947-e558-4909-8c6a-85a008823b1c\") " pod="openshift-monitoring/node-exporter-dpmd2"
Jul 28 22:09:44.373042 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:44.373007    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/59091947-e558-4909-8c6a-85a008823b1c-node-exporter-tls\") pod \"node-exporter-dpmd2\" (UID: \"59091947-e558-4909-8c6a-85a008823b1c\") " pod="openshift-monitoring/node-exporter-dpmd2"
Jul 28 22:09:44.375244 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:44.375225    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/59091947-e558-4909-8c6a-85a008823b1c-node-exporter-tls\") pod \"node-exporter-dpmd2\" (UID: \"59091947-e558-4909-8c6a-85a008823b1c\") " pod="openshift-monitoring/node-exporter-dpmd2"
Jul 28 22:09:44.569042 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:44.569016    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dpmd2"
Jul 28 22:09:44.581209 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:09:44.581183    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59091947_e558_4909_8c6a_85a008823b1c.slice/crio-218c22d4f8b63cd5d6ec0a6f6ef4716ee5057c824ff75de90b4e8b34be37fad3 WatchSource:0}: Error finding container 218c22d4f8b63cd5d6ec0a6f6ef4716ee5057c824ff75de90b4e8b34be37fad3: Status 404 returned error can't find the container with id 218c22d4f8b63cd5d6ec0a6f6ef4716ee5057c824ff75de90b4e8b34be37fad3
Jul 28 22:09:44.626905 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:44.626834    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-vqx8j" podUID="331ef64d-2011-4149-b38c-88921f08f078"
Jul 28 22:09:45.010511 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:45.010485    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vqx8j"
Jul 28 22:09:45.010898 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:45.010487    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dpmd2" event={"ID":"59091947-e558-4909-8c6a-85a008823b1c","Type":"ContainerStarted","Data":"218c22d4f8b63cd5d6ec0a6f6ef4716ee5057c824ff75de90b4e8b34be37fad3"}
Jul 28 22:09:45.079910 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:45.079878    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-747c5cb5b4-9qkkl"]
Jul 28 22:09:45.083866 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:45.083850    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-747c5cb5b4-9qkkl"
Jul 28 22:09:45.086659 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:45.086629    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 22:09:45.087833 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:45.087815    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 22:09:45.087922 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:45.087859    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-2m99s\""
Jul 28 22:09:45.087922 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:45.087859    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 22:09:45.088024 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:45.087952    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 22:09:45.088077 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:45.088004    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 22:09:45.088269 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:45.088254    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 22:09:45.088366 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:45.088335    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 22:09:45.092707 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:45.092683    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 28 22:09:45.095407 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:45.095380    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-747c5cb5b4-9qkkl"]
Jul 28 22:09:45.179649 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:45.179613    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e4d5a32-ec38-4246-aaea-49002e594290-trusted-ca-bundle\") pod \"console-747c5cb5b4-9qkkl\" (UID: \"2e4d5a32-ec38-4246-aaea-49002e594290\") " pod="openshift-console/console-747c5cb5b4-9qkkl"
Jul 28 22:09:45.179649 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:45.179646    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e4d5a32-ec38-4246-aaea-49002e594290-service-ca\") pod \"console-747c5cb5b4-9qkkl\" (UID: \"2e4d5a32-ec38-4246-aaea-49002e594290\") " pod="openshift-console/console-747c5cb5b4-9qkkl"
Jul 28 22:09:45.179903 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:45.179683    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e4d5a32-ec38-4246-aaea-49002e594290-oauth-serving-cert\") pod \"console-747c5cb5b4-9qkkl\" (UID: \"2e4d5a32-ec38-4246-aaea-49002e594290\") " pod="openshift-console/console-747c5cb5b4-9qkkl"
Jul 28 22:09:45.179903 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:45.179797    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e4d5a32-ec38-4246-aaea-49002e594290-console-serving-cert\") pod \"console-747c5cb5b4-9qkkl\" (UID: \"2e4d5a32-ec38-4246-aaea-49002e594290\") " pod="openshift-console/console-747c5cb5b4-9qkkl"
Jul 28 22:09:45.179903 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:45.179847    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpqtt\" (UniqueName: \"kubernetes.io/projected/2e4d5a32-ec38-4246-aaea-49002e594290-kube-api-access-kpqtt\") pod \"console-747c5cb5b4-9qkkl\" (UID: \"2e4d5a32-ec38-4246-aaea-49002e594290\") " pod="openshift-console/console-747c5cb5b4-9qkkl"
Jul 28 22:09:45.180029 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:45.179931    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e4d5a32-ec38-4246-aaea-49002e594290-console-oauth-config\") pod \"console-747c5cb5b4-9qkkl\" (UID: \"2e4d5a32-ec38-4246-aaea-49002e594290\") " pod="openshift-console/console-747c5cb5b4-9qkkl"
Jul 28 22:09:45.180029 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:45.179970    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e4d5a32-ec38-4246-aaea-49002e594290-console-config\") pod \"console-747c5cb5b4-9qkkl\" (UID: \"2e4d5a32-ec38-4246-aaea-49002e594290\") " pod="openshift-console/console-747c5cb5b4-9qkkl"
Jul 28 22:09:45.280921 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:45.280831    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e4d5a32-ec38-4246-aaea-49002e594290-trusted-ca-bundle\") pod \"console-747c5cb5b4-9qkkl\" (UID: \"2e4d5a32-ec38-4246-aaea-49002e594290\") " pod="openshift-console/console-747c5cb5b4-9qkkl"
Jul 28 22:09:45.280921 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:45.280884    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e4d5a32-ec38-4246-aaea-49002e594290-service-ca\") pod \"console-747c5cb5b4-9qkkl\" (UID: \"2e4d5a32-ec38-4246-aaea-49002e594290\") " pod="openshift-console/console-747c5cb5b4-9qkkl"
Jul 28 22:09:45.281210 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:45.280925    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e4d5a32-ec38-4246-aaea-49002e594290-oauth-serving-cert\") pod \"console-747c5cb5b4-9qkkl\" (UID: \"2e4d5a32-ec38-4246-aaea-49002e594290\") " pod="openshift-console/console-747c5cb5b4-9qkkl"
Jul 28 22:09:45.281210 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:45.280965    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e4d5a32-ec38-4246-aaea-49002e594290-console-serving-cert\") pod \"console-747c5cb5b4-9qkkl\" (UID: \"2e4d5a32-ec38-4246-aaea-49002e594290\") " pod="openshift-console/console-747c5cb5b4-9qkkl"
Jul 28 22:09:45.281210 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:45.280999    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpqtt\" (UniqueName: \"kubernetes.io/projected/2e4d5a32-ec38-4246-aaea-49002e594290-kube-api-access-kpqtt\") pod \"console-747c5cb5b4-9qkkl\" (UID: \"2e4d5a32-ec38-4246-aaea-49002e594290\") " pod="openshift-console/console-747c5cb5b4-9qkkl"
Jul 28 22:09:45.281210 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:45.281058    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e4d5a32-ec38-4246-aaea-49002e594290-console-oauth-config\") pod \"console-747c5cb5b4-9qkkl\" (UID: \"2e4d5a32-ec38-4246-aaea-49002e594290\") " pod="openshift-console/console-747c5cb5b4-9qkkl"
Jul 28 22:09:45.281210 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:45.281092    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e4d5a32-ec38-4246-aaea-49002e594290-console-config\") pod \"console-747c5cb5b4-9qkkl\" (UID: \"2e4d5a32-ec38-4246-aaea-49002e594290\") " pod="openshift-console/console-747c5cb5b4-9qkkl"
Jul 28 22:09:45.281791 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:45.281761    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e4d5a32-ec38-4246-aaea-49002e594290-oauth-serving-cert\") pod \"console-747c5cb5b4-9qkkl\" (UID: \"2e4d5a32-ec38-4246-aaea-49002e594290\") " pod="openshift-console/console-747c5cb5b4-9qkkl"
Jul 28 22:09:45.281898 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:45.281793    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e4d5a32-ec38-4246-aaea-49002e594290-service-ca\") pod \"console-747c5cb5b4-9qkkl\" (UID: \"2e4d5a32-ec38-4246-aaea-49002e594290\") " pod="openshift-console/console-747c5cb5b4-9qkkl"
Jul 28 22:09:45.281898 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:45.281829    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e4d5a32-ec38-4246-aaea-49002e594290-console-config\") pod \"console-747c5cb5b4-9qkkl\" (UID: \"2e4d5a32-ec38-4246-aaea-49002e594290\") " pod="openshift-console/console-747c5cb5b4-9qkkl"
Jul 28 22:09:45.281898 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:45.281853    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e4d5a32-ec38-4246-aaea-49002e594290-trusted-ca-bundle\") pod \"console-747c5cb5b4-9qkkl\" (UID: \"2e4d5a32-ec38-4246-aaea-49002e594290\") " pod="openshift-console/console-747c5cb5b4-9qkkl"
Jul 28 22:09:45.284148 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:45.284120    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e4d5a32-ec38-4246-aaea-49002e594290-console-serving-cert\") pod \"console-747c5cb5b4-9qkkl\" (UID: \"2e4d5a32-ec38-4246-aaea-49002e594290\") " pod="openshift-console/console-747c5cb5b4-9qkkl"
Jul 28 22:09:45.284370 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:45.284331    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e4d5a32-ec38-4246-aaea-49002e594290-console-oauth-config\") pod \"console-747c5cb5b4-9qkkl\" (UID: \"2e4d5a32-ec38-4246-aaea-49002e594290\") " pod="openshift-console/console-747c5cb5b4-9qkkl"
Jul 28 22:09:45.292792 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:45.292747    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpqtt\" (UniqueName: \"kubernetes.io/projected/2e4d5a32-ec38-4246-aaea-49002e594290-kube-api-access-kpqtt\") pod \"console-747c5cb5b4-9qkkl\" (UID: \"2e4d5a32-ec38-4246-aaea-49002e594290\") " pod="openshift-console/console-747c5cb5b4-9qkkl"
Jul 28 22:09:45.394236 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:45.394205    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-747c5cb5b4-9qkkl"
Jul 28 22:09:45.648243 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:45.648213    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-747c5cb5b4-9qkkl"]
Jul 28 22:09:45.651263 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:09:45.651237    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e4d5a32_ec38_4246_aaea_49002e594290.slice/crio-39812f5042818858b7543f48599879ab16a15fe261a287a0971443fbd3f97849 WatchSource:0}: Error finding container 39812f5042818858b7543f48599879ab16a15fe261a287a0971443fbd3f97849: Status 404 returned error can't find the container with id 39812f5042818858b7543f48599879ab16a15fe261a287a0971443fbd3f97849
Jul 28 22:09:46.016095 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:46.016062    2563 generic.go:356] "Generic (PLEG): container finished" podID="59091947-e558-4909-8c6a-85a008823b1c" containerID="58f3746d438cf6801ed2ed1df7b0ba79beb36514df3d78316b1e1cc732dfe508" exitCode=0
Jul 28 22:09:46.016558 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:46.016152    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dpmd2" event={"ID":"59091947-e558-4909-8c6a-85a008823b1c","Type":"ContainerDied","Data":"58f3746d438cf6801ed2ed1df7b0ba79beb36514df3d78316b1e1cc732dfe508"}
Jul 28 22:09:46.017653 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:46.017626    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-747c5cb5b4-9qkkl" event={"ID":"2e4d5a32-ec38-4246-aaea-49002e594290","Type":"ContainerStarted","Data":"39812f5042818858b7543f48599879ab16a15fe261a287a0971443fbd3f97849"}
Jul 28 22:09:47.023560 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:47.023529    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dpmd2" event={"ID":"59091947-e558-4909-8c6a-85a008823b1c","Type":"ContainerStarted","Data":"22c623f3bf060051455b15763b55dcb9831115d690488a8538677024e7b050e5"}
Jul 28 22:09:47.023986 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:47.023589    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dpmd2" event={"ID":"59091947-e558-4909-8c6a-85a008823b1c","Type":"ContainerStarted","Data":"47d73186f2f859ea14cedcf2bb143070983e542a44709f45eb70ab5e9ac2526f"}
Jul 28 22:09:47.060201 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:47.060136    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dpmd2" podStartSLOduration=3.011601226 podStartE2EDuration="4.060113781s" totalImagesPullingTime="983.462226ms" totalInitContainerRuntime="65.050329ms" isStatefulPod=true podCreationTimestamp="2026-07-28 22:09:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:09:47.059068494 +0000 UTC m=+284.439206140" watchObservedRunningTime="2026-07-28 22:09:47.060113781 +0000 UTC m=+284.440251426"
Jul 28 22:09:47.098930 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:47.098902    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf"
Jul 28 22:09:48.007783 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:48.007749    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/331ef64d-2011-4149-b38c-88921f08f078-metrics-tls\") pod \"dns-default-vqx8j\" (UID: \"331ef64d-2011-4149-b38c-88921f08f078\") " pod="openshift-dns/dns-default-vqx8j"
Jul 28 22:09:48.010461 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:48.010432    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/331ef64d-2011-4149-b38c-88921f08f078-metrics-tls\") pod \"dns-default-vqx8j\" (UID: \"331ef64d-2011-4149-b38c-88921f08f078\") " pod="openshift-dns/dns-default-vqx8j"
Jul 28 22:09:48.013918 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:48.013891    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rvh82\""
Jul 28 22:09:48.022108 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:48.022083    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vqx8j"
Jul 28 22:09:48.121239 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:48.121208    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-77cbc6f9d4-hxcn7"]
Jul 28 22:09:48.125977 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:48.125953    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-77cbc6f9d4-hxcn7"
Jul 28 22:09:48.130275 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:48.130251    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 28 22:09:48.131441 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:48.131418    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 28 22:09:48.131554 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:48.131437    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 28 22:09:48.131554 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:48.131418    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-cfurds4tb2j82\""
Jul 28 22:09:48.131670 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:48.131556    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-kgn7p\""
Jul 28 22:09:48.131670 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:48.131574    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 22:09:48.140278 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:48.140256    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-77cbc6f9d4-hxcn7"]
Jul 28 22:09:48.209400 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:48.209362    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wvdd\" (UniqueName: \"kubernetes.io/projected/42c93fdc-261d-46e3-82f6-89d82c18df1b-kube-api-access-8wvdd\") pod \"metrics-server-77cbc6f9d4-hxcn7\" (UID: \"42c93fdc-261d-46e3-82f6-89d82c18df1b\") " pod="openshift-monitoring/metrics-server-77cbc6f9d4-hxcn7"
Jul 28 22:09:48.209582 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:48.209492    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42c93fdc-261d-46e3-82f6-89d82c18df1b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-77cbc6f9d4-hxcn7\" (UID: \"42c93fdc-261d-46e3-82f6-89d82c18df1b\") " pod="openshift-monitoring/metrics-server-77cbc6f9d4-hxcn7"
Jul 28 22:09:48.209582 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:48.209536    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/42c93fdc-261d-46e3-82f6-89d82c18df1b-secret-metrics-server-client-certs\") pod \"metrics-server-77cbc6f9d4-hxcn7\" (UID: \"42c93fdc-261d-46e3-82f6-89d82c18df1b\") " pod="openshift-monitoring/metrics-server-77cbc6f9d4-hxcn7"
Jul 28 22:09:48.209582 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:48.209569    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/42c93fdc-261d-46e3-82f6-89d82c18df1b-secret-metrics-server-tls\") pod \"metrics-server-77cbc6f9d4-hxcn7\" (UID: \"42c93fdc-261d-46e3-82f6-89d82c18df1b\") " pod="openshift-monitoring/metrics-server-77cbc6f9d4-hxcn7"
Jul 28 22:09:48.209723 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:48.209591    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/42c93fdc-261d-46e3-82f6-89d82c18df1b-client-ca-bundle\") pod \"metrics-server-77cbc6f9d4-hxcn7\" (UID: \"42c93fdc-261d-46e3-82f6-89d82c18df1b\") " pod="openshift-monitoring/metrics-server-77cbc6f9d4-hxcn7"
Jul 28 22:09:48.209760 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:48.209728    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/42c93fdc-261d-46e3-82f6-89d82c18df1b-audit-log\") pod \"metrics-server-77cbc6f9d4-hxcn7\" (UID: \"42c93fdc-261d-46e3-82f6-89d82c18df1b\") " pod="openshift-monitoring/metrics-server-77cbc6f9d4-hxcn7"
Jul 28 22:09:48.209813 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:48.209782    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/42c93fdc-261d-46e3-82f6-89d82c18df1b-metrics-server-audit-profiles\") pod \"metrics-server-77cbc6f9d4-hxcn7\" (UID: \"42c93fdc-261d-46e3-82f6-89d82c18df1b\") " pod="openshift-monitoring/metrics-server-77cbc6f9d4-hxcn7"
Jul 28 22:09:48.310732 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:48.310643    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/42c93fdc-261d-46e3-82f6-89d82c18df1b-audit-log\") pod \"metrics-server-77cbc6f9d4-hxcn7\" (UID: \"42c93fdc-261d-46e3-82f6-89d82c18df1b\") " pod="openshift-monitoring/metrics-server-77cbc6f9d4-hxcn7"
Jul 28 22:09:48.310732 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:48.310690    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/42c93fdc-261d-46e3-82f6-89d82c18df1b-metrics-server-audit-profiles\") pod \"metrics-server-77cbc6f9d4-hxcn7\" (UID: \"42c93fdc-261d-46e3-82f6-89d82c18df1b\") " pod="openshift-monitoring/metrics-server-77cbc6f9d4-hxcn7"
Jul 28 22:09:48.310732 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:48.310712    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wvdd\" (UniqueName: \"kubernetes.io/projected/42c93fdc-261d-46e3-82f6-89d82c18df1b-kube-api-access-8wvdd\") pod \"metrics-server-77cbc6f9d4-hxcn7\" (UID: \"42c93fdc-261d-46e3-82f6-89d82c18df1b\") " pod="openshift-monitoring/metrics-server-77cbc6f9d4-hxcn7"
Jul 28 22:09:48.311004 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:48.310783    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42c93fdc-261d-46e3-82f6-89d82c18df1b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-77cbc6f9d4-hxcn7\" (UID: \"42c93fdc-261d-46e3-82f6-89d82c18df1b\") " pod="openshift-monitoring/metrics-server-77cbc6f9d4-hxcn7"
Jul 28 22:09:48.311004 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:48.310835    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/42c93fdc-261d-46e3-82f6-89d82c18df1b-secret-metrics-server-client-certs\") pod \"metrics-server-77cbc6f9d4-hxcn7\" (UID: \"42c93fdc-261d-46e3-82f6-89d82c18df1b\") " pod="openshift-monitoring/metrics-server-77cbc6f9d4-hxcn7"
Jul 28 22:09:48.311004 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:48.310883    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/42c93fdc-261d-46e3-82f6-89d82c18df1b-secret-metrics-server-tls\") pod \"metrics-server-77cbc6f9d4-hxcn7\" (UID: \"42c93fdc-261d-46e3-82f6-89d82c18df1b\") " pod="openshift-monitoring/metrics-server-77cbc6f9d4-hxcn7"
Jul 28 22:09:48.311004 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:48.310906    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/42c93fdc-261d-46e3-82f6-89d82c18df1b-client-ca-bundle\") pod \"metrics-server-77cbc6f9d4-hxcn7\" (UID: \"42c93fdc-261d-46e3-82f6-89d82c18df1b\") " pod="openshift-monitoring/metrics-server-77cbc6f9d4-hxcn7"
Jul 28 22:09:48.311211 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:48.311140    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/42c93fdc-261d-46e3-82f6-89d82c18df1b-audit-log\") pod \"metrics-server-77cbc6f9d4-hxcn7\" (UID: \"42c93fdc-261d-46e3-82f6-89d82c18df1b\") " pod="openshift-monitoring/metrics-server-77cbc6f9d4-hxcn7"
Jul 28 22:09:48.311857 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:48.311824    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/42c93fdc-261d-46e3-82f6-89d82c18df1b-metrics-server-audit-profiles\") pod \"metrics-server-77cbc6f9d4-hxcn7\" (UID: \"42c93fdc-261d-46e3-82f6-89d82c18df1b\") " pod="openshift-monitoring/metrics-server-77cbc6f9d4-hxcn7"
Jul 28 22:09:48.311984 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:48.311833    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42c93fdc-261d-46e3-82f6-89d82c18df1b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-77cbc6f9d4-hxcn7\" (UID: \"42c93fdc-261d-46e3-82f6-89d82c18df1b\") " pod="openshift-monitoring/metrics-server-77cbc6f9d4-hxcn7"
Jul 28 22:09:48.313600 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:48.313576    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/42c93fdc-261d-46e3-82f6-89d82c18df1b-secret-metrics-server-client-certs\") pod \"metrics-server-77cbc6f9d4-hxcn7\" (UID: \"42c93fdc-261d-46e3-82f6-89d82c18df1b\") " pod="openshift-monitoring/metrics-server-77cbc6f9d4-hxcn7"
Jul 28 22:09:48.313722 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:48.313708    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/42c93fdc-261d-46e3-82f6-89d82c18df1b-client-ca-bundle\") pod \"metrics-server-77cbc6f9d4-hxcn7\" (UID: \"42c93fdc-261d-46e3-82f6-89d82c18df1b\") " pod="openshift-monitoring/metrics-server-77cbc6f9d4-hxcn7"
Jul 28 22:09:48.313784 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:48.313740    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/42c93fdc-261d-46e3-82f6-89d82c18df1b-secret-metrics-server-tls\") pod \"metrics-server-77cbc6f9d4-hxcn7\" (UID: \"42c93fdc-261d-46e3-82f6-89d82c18df1b\") " pod="openshift-monitoring/metrics-server-77cbc6f9d4-hxcn7"
Jul 28 22:09:48.323110 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:48.323089    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wvdd\" (UniqueName: \"kubernetes.io/projected/42c93fdc-261d-46e3-82f6-89d82c18df1b-kube-api-access-8wvdd\") pod \"metrics-server-77cbc6f9d4-hxcn7\" (UID: \"42c93fdc-261d-46e3-82f6-89d82c18df1b\") " pod="openshift-monitoring/metrics-server-77cbc6f9d4-hxcn7"
Jul 28 22:09:48.436730 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:48.436682    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-77cbc6f9d4-hxcn7"
Jul 28 22:09:48.487969 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:48.487858    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vqx8j"]
Jul 28 22:09:48.582268 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:48.581868    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-77cbc6f9d4-hxcn7"]
Jul 28 22:09:48.586803 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:09:48.586774    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42c93fdc_261d_46e3_82f6_89d82c18df1b.slice/crio-d969fb5622e28ff31a92adda2bcf64d63aaef7d6c5d638bb668a6556960950b8 WatchSource:0}: Error finding container d969fb5622e28ff31a92adda2bcf64d63aaef7d6c5d638bb668a6556960950b8: Status 404 returned error can't find the container with id d969fb5622e28ff31a92adda2bcf64d63aaef7d6c5d638bb668a6556960950b8
Jul 28 22:09:49.032760 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:49.032729    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-77cbc6f9d4-hxcn7" event={"ID":"42c93fdc-261d-46e3-82f6-89d82c18df1b","Type":"ContainerStarted","Data":"d969fb5622e28ff31a92adda2bcf64d63aaef7d6c5d638bb668a6556960950b8"}
Jul 28 22:09:49.034026 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:49.033988    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vqx8j" event={"ID":"331ef64d-2011-4149-b38c-88921f08f078","Type":"ContainerStarted","Data":"f5b00e91974c853a2d5bef646bdbdcfa5a2bf3eaa76aa5ae8cb21dcafbab59a8"}
Jul 28 22:09:49.035629 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:49.035602    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-747c5cb5b4-9qkkl" event={"ID":"2e4d5a32-ec38-4246-aaea-49002e594290","Type":"ContainerStarted","Data":"5fed2baa11b6d2b7e602e90377c6cceb181b13494dc2eac64423e0d3600080ef"}
Jul 28 22:09:49.064240 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:49.064183    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-747c5cb5b4-9qkkl" podStartSLOduration=1.302911544 podStartE2EDuration="4.064164924s" totalImagesPullingTime="2.76125338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:09:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:09:49.064079151 +0000 UTC m=+286.444216798" watchObservedRunningTime="2026-07-28 22:09:49.064164924 +0000 UTC m=+286.444302569"
Jul 28 22:09:49.948801 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:49.948766    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-86664d9944-5hsqd"
Jul 28 22:09:51.045329 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:51.045293    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vqx8j" event={"ID":"331ef64d-2011-4149-b38c-88921f08f078","Type":"ContainerStarted","Data":"b32cd665cbc9e35c90f3bdec850e15d2ce0d58d1d3f93776ce7c6ecf47b6ed31"}
Jul 28 22:09:51.045800 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:51.045385    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vqx8j"
Jul 28 22:09:51.045800 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:51.045410    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vqx8j" event={"ID":"331ef64d-2011-4149-b38c-88921f08f078","Type":"ContainerStarted","Data":"4159c6ecc1e426689d08503deeb20d086ea6cd261fad79dd9cfcba3ba5b811f9"}
Jul 28 22:09:51.046732 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:51.046710    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-77cbc6f9d4-hxcn7" event={"ID":"42c93fdc-261d-46e3-82f6-89d82c18df1b","Type":"ContainerStarted","Data":"fc33caba4bd2cc08331f18a97f1504279ef721251ba7efb7a0e49769ab5b0a28"}
Jul 28 22:09:51.064053 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:51.064001    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vqx8j" podStartSLOduration=250.876751045 podStartE2EDuration="4m13.063985463s" totalImagesPullingTime="2.187234418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:05:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:09:51.062821323 +0000 UTC m=+288.442958966" watchObservedRunningTime="2026-07-28 22:09:51.063985463 +0000 UTC m=+288.444123107"
Jul 28 22:09:51.081754 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:51.081704    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-77cbc6f9d4-hxcn7" podStartSLOduration=0.991736898 podStartE2EDuration="3.081688936s" totalImagesPullingTime="2.089952038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:09:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:09:51.081116164 +0000 UTC m=+288.461253809" watchObservedRunningTime="2026-07-28 22:09:51.081688936 +0000 UTC m=+288.461826580"
Jul 28 22:09:52.110866 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:52.110824    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf" podUID="edcb52a8-15ce-419d-93dc-cc405f75076e" containerName="registry" containerID="cri-o://172601c8d016666801d9f5aa3e7d600e84aba90fdc641cd9a5dba90f6f4c00ca" gracePeriod=30
Jul 28 22:09:52.344745 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:52.344724    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf"
Jul 28 22:09:52.451010 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:52.450978    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/edcb52a8-15ce-419d-93dc-cc405f75076e-trusted-ca\") pod \"edcb52a8-15ce-419d-93dc-cc405f75076e\" (UID: \"edcb52a8-15ce-419d-93dc-cc405f75076e\") "
Jul 28 22:09:52.451211 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:52.451052    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/edcb52a8-15ce-419d-93dc-cc405f75076e-image-registry-private-configuration\") pod \"edcb52a8-15ce-419d-93dc-cc405f75076e\" (UID: \"edcb52a8-15ce-419d-93dc-cc405f75076e\") "
Jul 28 22:09:52.451211 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:52.451082    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/edcb52a8-15ce-419d-93dc-cc405f75076e-installation-pull-secrets\") pod \"edcb52a8-15ce-419d-93dc-cc405f75076e\" (UID: \"edcb52a8-15ce-419d-93dc-cc405f75076e\") "
Jul 28 22:09:52.451211 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:52.451120    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/edcb52a8-15ce-419d-93dc-cc405f75076e-registry-tls\") pod \"edcb52a8-15ce-419d-93dc-cc405f75076e\" (UID: \"edcb52a8-15ce-419d-93dc-cc405f75076e\") "
Jul 28 22:09:52.451376 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:52.451314    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdsjj\" (UniqueName: \"kubernetes.io/projected/edcb52a8-15ce-419d-93dc-cc405f75076e-kube-api-access-hdsjj\") pod \"edcb52a8-15ce-419d-93dc-cc405f75076e\" (UID: \"edcb52a8-15ce-419d-93dc-cc405f75076e\") "
Jul 28 22:09:52.451437 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:52.451400    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/edcb52a8-15ce-419d-93dc-cc405f75076e-registry-certificates\") pod \"edcb52a8-15ce-419d-93dc-cc405f75076e\" (UID: \"edcb52a8-15ce-419d-93dc-cc405f75076e\") "
Jul 28 22:09:52.451496 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:52.451434    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/edcb52a8-15ce-419d-93dc-cc405f75076e-ca-trust-extracted\") pod \"edcb52a8-15ce-419d-93dc-cc405f75076e\" (UID: \"edcb52a8-15ce-419d-93dc-cc405f75076e\") "
Jul 28 22:09:52.451496 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:52.451465    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/edcb52a8-15ce-419d-93dc-cc405f75076e-bound-sa-token\") pod \"edcb52a8-15ce-419d-93dc-cc405f75076e\" (UID: \"edcb52a8-15ce-419d-93dc-cc405f75076e\") "
Jul 28 22:09:52.451840 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:52.451777    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edcb52a8-15ce-419d-93dc-cc405f75076e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "edcb52a8-15ce-419d-93dc-cc405f75076e" (UID: "edcb52a8-15ce-419d-93dc-cc405f75076e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:09:52.453391 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:52.453368    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edcb52a8-15ce-419d-93dc-cc405f75076e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "edcb52a8-15ce-419d-93dc-cc405f75076e" (UID: "edcb52a8-15ce-419d-93dc-cc405f75076e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:09:52.454218 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:52.454174    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edcb52a8-15ce-419d-93dc-cc405f75076e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "edcb52a8-15ce-419d-93dc-cc405f75076e" (UID: "edcb52a8-15ce-419d-93dc-cc405f75076e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:09:52.454385 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:52.454281    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edcb52a8-15ce-419d-93dc-cc405f75076e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "edcb52a8-15ce-419d-93dc-cc405f75076e" (UID: "edcb52a8-15ce-419d-93dc-cc405f75076e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:09:52.454385 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:52.454294    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edcb52a8-15ce-419d-93dc-cc405f75076e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "edcb52a8-15ce-419d-93dc-cc405f75076e" (UID: "edcb52a8-15ce-419d-93dc-cc405f75076e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:09:52.454385 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:52.454373    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edcb52a8-15ce-419d-93dc-cc405f75076e-kube-api-access-hdsjj" (OuterVolumeSpecName: "kube-api-access-hdsjj") pod "edcb52a8-15ce-419d-93dc-cc405f75076e" (UID: "edcb52a8-15ce-419d-93dc-cc405f75076e"). InnerVolumeSpecName "kube-api-access-hdsjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:09:52.454735 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:52.454714    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edcb52a8-15ce-419d-93dc-cc405f75076e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "edcb52a8-15ce-419d-93dc-cc405f75076e" (UID: "edcb52a8-15ce-419d-93dc-cc405f75076e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:09:52.463605 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:52.463577    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edcb52a8-15ce-419d-93dc-cc405f75076e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "edcb52a8-15ce-419d-93dc-cc405f75076e" (UID: "edcb52a8-15ce-419d-93dc-cc405f75076e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:09:52.552728 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:52.552703    2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/edcb52a8-15ce-419d-93dc-cc405f75076e-registry-certificates\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:09:52.552728 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:52.552725    2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/edcb52a8-15ce-419d-93dc-cc405f75076e-ca-trust-extracted\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:09:52.552881 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:52.552735    2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/edcb52a8-15ce-419d-93dc-cc405f75076e-bound-sa-token\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:09:52.552881 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:52.552745    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/edcb52a8-15ce-419d-93dc-cc405f75076e-trusted-ca\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:09:52.552881 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:52.552753    2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/edcb52a8-15ce-419d-93dc-cc405f75076e-image-registry-private-configuration\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:09:52.552881 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:52.552762    2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/edcb52a8-15ce-419d-93dc-cc405f75076e-installation-pull-secrets\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:09:52.552881 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:52.552771    2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/edcb52a8-15ce-419d-93dc-cc405f75076e-registry-tls\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:09:52.552881 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:52.552780    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdsjj\" (UniqueName: \"kubernetes.io/projected/edcb52a8-15ce-419d-93dc-cc405f75076e-kube-api-access-hdsjj\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:09:53.055413 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:53.055378    2563 generic.go:356] "Generic (PLEG): container finished" podID="edcb52a8-15ce-419d-93dc-cc405f75076e" containerID="172601c8d016666801d9f5aa3e7d600e84aba90fdc641cd9a5dba90f6f4c00ca" exitCode=0
Jul 28 22:09:53.055582 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:53.055420    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf"
Jul 28 22:09:53.055582 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:53.055444    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf" event={"ID":"edcb52a8-15ce-419d-93dc-cc405f75076e","Type":"ContainerDied","Data":"172601c8d016666801d9f5aa3e7d600e84aba90fdc641cd9a5dba90f6f4c00ca"}
Jul 28 22:09:53.055582 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:53.055520    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f9c64b766-fwkvf" event={"ID":"edcb52a8-15ce-419d-93dc-cc405f75076e","Type":"ContainerDied","Data":"c9cc94d946a9476810a1430e4872c581f78335d8cf08af0b5f6b4ee0bec91979"}
Jul 28 22:09:53.055582 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:53.055551    2563 scope.go:117] "RemoveContainer" containerID="172601c8d016666801d9f5aa3e7d600e84aba90fdc641cd9a5dba90f6f4c00ca"
Jul 28 22:09:53.063981 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:53.063960    2563 scope.go:117] "RemoveContainer" containerID="172601c8d016666801d9f5aa3e7d600e84aba90fdc641cd9a5dba90f6f4c00ca"
Jul 28 22:09:53.064227 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:53.064208    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"172601c8d016666801d9f5aa3e7d600e84aba90fdc641cd9a5dba90f6f4c00ca\": container with ID starting with 172601c8d016666801d9f5aa3e7d600e84aba90fdc641cd9a5dba90f6f4c00ca not found: ID does not exist" containerID="172601c8d016666801d9f5aa3e7d600e84aba90fdc641cd9a5dba90f6f4c00ca"
Jul 28 22:09:53.064290 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:53.064235    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"172601c8d016666801d9f5aa3e7d600e84aba90fdc641cd9a5dba90f6f4c00ca"} err="failed to get container status \"172601c8d016666801d9f5aa3e7d600e84aba90fdc641cd9a5dba90f6f4c00ca\": rpc error: code = NotFound desc = could not find container \"172601c8d016666801d9f5aa3e7d600e84aba90fdc641cd9a5dba90f6f4c00ca\": container with ID starting with 172601c8d016666801d9f5aa3e7d600e84aba90fdc641cd9a5dba90f6f4c00ca not found: ID does not exist"
Jul 28 22:09:53.076983 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:53.076961    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f9c64b766-fwkvf"]
Jul 28 22:09:53.081702 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:53.081680    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7f9c64b766-fwkvf"]
Jul 28 22:09:53.128273 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:53.128246    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edcb52a8-15ce-419d-93dc-cc405f75076e" path="/var/lib/kubelet/pods/edcb52a8-15ce-419d-93dc-cc405f75076e/volumes"
Jul 28 22:09:54.963106 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:54.963071    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-747c5cb5b4-9qkkl"]
Jul 28 22:09:54.998562 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:54.998531    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-c57655f7f-nsfd4"]
Jul 28 22:09:54.998783 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:54.998771    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edcb52a8-15ce-419d-93dc-cc405f75076e" containerName="registry"
Jul 28 22:09:54.998842 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:54.998784    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="edcb52a8-15ce-419d-93dc-cc405f75076e" containerName="registry"
Jul 28 22:09:54.998899 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:54.998847    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="edcb52a8-15ce-419d-93dc-cc405f75076e" containerName="registry"
Jul 28 22:09:55.001504 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:55.001488    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c57655f7f-nsfd4"
Jul 28 22:09:55.015107 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:55.015085    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c57655f7f-nsfd4"]
Jul 28 22:09:55.074887 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:55.074857    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-trusted-ca-bundle\") pod \"console-c57655f7f-nsfd4\" (UID: \"fd8a3086-bf14-4ad1-bf98-f8fa765aceab\") " pod="openshift-console/console-c57655f7f-nsfd4"
Jul 28 22:09:55.074887 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:55.074893    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-console-serving-cert\") pod \"console-c57655f7f-nsfd4\" (UID: \"fd8a3086-bf14-4ad1-bf98-f8fa765aceab\") " pod="openshift-console/console-c57655f7f-nsfd4"
Jul 28 22:09:55.075086 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:55.074937    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p2xv\" (UniqueName: \"kubernetes.io/projected/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-kube-api-access-8p2xv\") pod \"console-c57655f7f-nsfd4\" (UID: \"fd8a3086-bf14-4ad1-bf98-f8fa765aceab\") " pod="openshift-console/console-c57655f7f-nsfd4"
Jul 28 22:09:55.075086 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:55.074963    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-console-config\") pod \"console-c57655f7f-nsfd4\" (UID: \"fd8a3086-bf14-4ad1-bf98-f8fa765aceab\") " pod="openshift-console/console-c57655f7f-nsfd4"
Jul 28 22:09:55.075086 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:55.074982    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-console-oauth-config\") pod \"console-c57655f7f-nsfd4\" (UID: \"fd8a3086-bf14-4ad1-bf98-f8fa765aceab\") " pod="openshift-console/console-c57655f7f-nsfd4"
Jul 28 22:09:55.075086 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:55.074997    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-oauth-serving-cert\") pod \"console-c57655f7f-nsfd4\" (UID: \"fd8a3086-bf14-4ad1-bf98-f8fa765aceab\") " pod="openshift-console/console-c57655f7f-nsfd4"
Jul 28 22:09:55.075213 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:55.075101    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-service-ca\") pod \"console-c57655f7f-nsfd4\" (UID: \"fd8a3086-bf14-4ad1-bf98-f8fa765aceab\") " pod="openshift-console/console-c57655f7f-nsfd4"
Jul 28 22:09:55.176143 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:55.176110    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-console-serving-cert\") pod \"console-c57655f7f-nsfd4\" (UID: \"fd8a3086-bf14-4ad1-bf98-f8fa765aceab\") " pod="openshift-console/console-c57655f7f-nsfd4"
Jul 28 22:09:55.176309 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:55.176165    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p2xv\" (UniqueName: \"kubernetes.io/projected/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-kube-api-access-8p2xv\") pod \"console-c57655f7f-nsfd4\" (UID: \"fd8a3086-bf14-4ad1-bf98-f8fa765aceab\") " pod="openshift-console/console-c57655f7f-nsfd4"
Jul 28 22:09:55.176309 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:55.176193    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-console-config\") pod \"console-c57655f7f-nsfd4\" (UID: \"fd8a3086-bf14-4ad1-bf98-f8fa765aceab\") " pod="openshift-console/console-c57655f7f-nsfd4"
Jul 28 22:09:55.176309 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:55.176212    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-console-oauth-config\") pod \"console-c57655f7f-nsfd4\" (UID: \"fd8a3086-bf14-4ad1-bf98-f8fa765aceab\") " pod="openshift-console/console-c57655f7f-nsfd4"
Jul 28 22:09:55.176309 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:55.176230    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-oauth-serving-cert\") pod \"console-c57655f7f-nsfd4\" (UID: \"fd8a3086-bf14-4ad1-bf98-f8fa765aceab\") " pod="openshift-console/console-c57655f7f-nsfd4"
Jul 28 22:09:55.176498 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:55.176411    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-service-ca\") pod \"console-c57655f7f-nsfd4\" (UID: \"fd8a3086-bf14-4ad1-bf98-f8fa765aceab\") " pod="openshift-console/console-c57655f7f-nsfd4"
Jul 28 22:09:55.176498 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:55.176448    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-trusted-ca-bundle\") pod \"console-c57655f7f-nsfd4\" (UID: \"fd8a3086-bf14-4ad1-bf98-f8fa765aceab\") " pod="openshift-console/console-c57655f7f-nsfd4"
Jul 28 22:09:55.176996 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:55.176968    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-console-config\") pod \"console-c57655f7f-nsfd4\" (UID: \"fd8a3086-bf14-4ad1-bf98-f8fa765aceab\") " pod="openshift-console/console-c57655f7f-nsfd4"
Jul 28 22:09:55.177133 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:55.177098    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-oauth-serving-cert\") pod \"console-c57655f7f-nsfd4\" (UID: \"fd8a3086-bf14-4ad1-bf98-f8fa765aceab\") " pod="openshift-console/console-c57655f7f-nsfd4"
Jul 28 22:09:55.177231 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:55.177212    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-trusted-ca-bundle\") pod \"console-c57655f7f-nsfd4\" (UID: \"fd8a3086-bf14-4ad1-bf98-f8fa765aceab\") " pod="openshift-console/console-c57655f7f-nsfd4"
Jul 28 22:09:55.177286 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:55.177212    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-service-ca\") pod \"console-c57655f7f-nsfd4\" (UID: \"fd8a3086-bf14-4ad1-bf98-f8fa765aceab\") " pod="openshift-console/console-c57655f7f-nsfd4"
Jul 28 22:09:55.178703 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:55.178678    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-console-serving-cert\") pod \"console-c57655f7f-nsfd4\" (UID: \"fd8a3086-bf14-4ad1-bf98-f8fa765aceab\") " pod="openshift-console/console-c57655f7f-nsfd4"
Jul 28 22:09:55.178788 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:55.178702    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-console-oauth-config\") pod \"console-c57655f7f-nsfd4\" (UID: \"fd8a3086-bf14-4ad1-bf98-f8fa765aceab\") " pod="openshift-console/console-c57655f7f-nsfd4"
Jul 28 22:09:55.185788 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:55.185768    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p2xv\" (UniqueName: \"kubernetes.io/projected/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-kube-api-access-8p2xv\") pod \"console-c57655f7f-nsfd4\" (UID: \"fd8a3086-bf14-4ad1-bf98-f8fa765aceab\") " pod="openshift-console/console-c57655f7f-nsfd4"
Jul 28 22:09:55.309743 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:55.309664    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c57655f7f-nsfd4"
Jul 28 22:09:55.394618 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:55.394585    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-747c5cb5b4-9qkkl"
Jul 28 22:09:55.444461 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:55.444306    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c57655f7f-nsfd4"]
Jul 28 22:09:55.446939 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:09:55.446914    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd8a3086_bf14_4ad1_bf98_f8fa765aceab.slice/crio-640454a87b2cd2a27b489f144d1a0685467e1247d69a2ebe5e37ba4ad1eddd06 WatchSource:0}: Error finding container 640454a87b2cd2a27b489f144d1a0685467e1247d69a2ebe5e37ba4ad1eddd06: Status 404 returned error can't find the container with id 640454a87b2cd2a27b489f144d1a0685467e1247d69a2ebe5e37ba4ad1eddd06
Jul 28 22:09:56.068428 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:56.068395    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c57655f7f-nsfd4" event={"ID":"fd8a3086-bf14-4ad1-bf98-f8fa765aceab","Type":"ContainerStarted","Data":"21d9e8cff459a20c5e035c882b9b47098f67af7cbc07a52209b60c174270fabc"}
Jul 28 22:09:56.068805 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:56.068440    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c57655f7f-nsfd4" event={"ID":"fd8a3086-bf14-4ad1-bf98-f8fa765aceab","Type":"ContainerStarted","Data":"640454a87b2cd2a27b489f144d1a0685467e1247d69a2ebe5e37ba4ad1eddd06"}
Jul 28 22:09:56.089553 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:09:56.089505    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-c57655f7f-nsfd4" podStartSLOduration=2.089489128 podStartE2EDuration="2.089489128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:09:56.087831352 +0000 UTC m=+293.467968997" watchObservedRunningTime="2026-07-28 22:09:56.089489128 +0000 UTC m=+293.469626771"
Jul 28 22:09:56.126297 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:09:56.126262    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-4xqhf" podUID="2f33e531-3341-4c01-940e-91f17519976c"
Jul 28 22:10:01.057094 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:01.057062    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vqx8j"
Jul 28 22:10:02.985871 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:02.985844    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mcj8p_7bab32b0-13c6-4055-916e-a7454bafb5b6/ovn-acl-logging/0.log"
Jul 28 22:10:02.986385 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:02.986346    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mcj8p_7bab32b0-13c6-4055-916e-a7454bafb5b6/ovn-acl-logging/0.log"
Jul 28 22:10:02.992670 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:02.992650    2563 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 22:10:05.310419 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:05.310380    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c57655f7f-nsfd4"
Jul 28 22:10:05.312844 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:05.310439    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c57655f7f-nsfd4"
Jul 28 22:10:05.315221 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:05.315200    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c57655f7f-nsfd4"
Jul 28 22:10:06.107076 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:06.107048    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c57655f7f-nsfd4"
Jul 28 22:10:08.437271 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:08.437235    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-77cbc6f9d4-hxcn7"
Jul 28 22:10:08.437711 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:08.437298    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-77cbc6f9d4-hxcn7"
Jul 28 22:10:10.125035 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:10.125006    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4xqhf"
Jul 28 22:10:19.981578 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:19.981533    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-747c5cb5b4-9qkkl" podUID="2e4d5a32-ec38-4246-aaea-49002e594290" containerName="console" containerID="cri-o://5fed2baa11b6d2b7e602e90377c6cceb181b13494dc2eac64423e0d3600080ef" gracePeriod=15
Jul 28 22:10:20.149787 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:20.149761    2563 generic.go:356] "Generic (PLEG): container finished" podID="2e4d5a32-ec38-4246-aaea-49002e594290" containerID="5fed2baa11b6d2b7e602e90377c6cceb181b13494dc2eac64423e0d3600080ef" exitCode=0
Jul 28 22:10:20.149891 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:20.149802    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-747c5cb5b4-9qkkl" event={"ID":"2e4d5a32-ec38-4246-aaea-49002e594290","Type":"ContainerDied","Data":"5fed2baa11b6d2b7e602e90377c6cceb181b13494dc2eac64423e0d3600080ef"}
Jul 28 22:10:20.215018 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:20.214996    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-747c5cb5b4-9qkkl"
Jul 28 22:10:20.286307 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:20.286236    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e4d5a32-ec38-4246-aaea-49002e594290-oauth-serving-cert\") pod \"2e4d5a32-ec38-4246-aaea-49002e594290\" (UID: \"2e4d5a32-ec38-4246-aaea-49002e594290\") "
Jul 28 22:10:20.286307 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:20.286269    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e4d5a32-ec38-4246-aaea-49002e594290-service-ca\") pod \"2e4d5a32-ec38-4246-aaea-49002e594290\" (UID: \"2e4d5a32-ec38-4246-aaea-49002e594290\") "
Jul 28 22:10:20.286307 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:20.286295    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e4d5a32-ec38-4246-aaea-49002e594290-console-config\") pod \"2e4d5a32-ec38-4246-aaea-49002e594290\" (UID: \"2e4d5a32-ec38-4246-aaea-49002e594290\") "
Jul 28 22:10:20.286593 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:20.286313    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e4d5a32-ec38-4246-aaea-49002e594290-trusted-ca-bundle\") pod \"2e4d5a32-ec38-4246-aaea-49002e594290\" (UID: \"2e4d5a32-ec38-4246-aaea-49002e594290\") "
Jul 28 22:10:20.286593 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:20.286346    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kpqtt\" (UniqueName: \"kubernetes.io/projected/2e4d5a32-ec38-4246-aaea-49002e594290-kube-api-access-kpqtt\") pod \"2e4d5a32-ec38-4246-aaea-49002e594290\" (UID: \"2e4d5a32-ec38-4246-aaea-49002e594290\") "
Jul 28 22:10:20.286593 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:20.286394    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e4d5a32-ec38-4246-aaea-49002e594290-console-serving-cert\") pod \"2e4d5a32-ec38-4246-aaea-49002e594290\" (UID: \"2e4d5a32-ec38-4246-aaea-49002e594290\") "
Jul 28 22:10:20.286593 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:20.286414    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e4d5a32-ec38-4246-aaea-49002e594290-console-oauth-config\") pod \"2e4d5a32-ec38-4246-aaea-49002e594290\" (UID: \"2e4d5a32-ec38-4246-aaea-49002e594290\") "
Jul 28 22:10:20.286822 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:20.286797    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e4d5a32-ec38-4246-aaea-49002e594290-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2e4d5a32-ec38-4246-aaea-49002e594290" (UID: "2e4d5a32-ec38-4246-aaea-49002e594290"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:10:20.287202 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:20.286969    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e4d5a32-ec38-4246-aaea-49002e594290-console-config" (OuterVolumeSpecName: "console-config") pod "2e4d5a32-ec38-4246-aaea-49002e594290" (UID: "2e4d5a32-ec38-4246-aaea-49002e594290"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:10:20.287202 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:20.287034    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e4d5a32-ec38-4246-aaea-49002e594290-service-ca" (OuterVolumeSpecName: "service-ca") pod "2e4d5a32-ec38-4246-aaea-49002e594290" (UID: "2e4d5a32-ec38-4246-aaea-49002e594290"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:10:20.287388 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:20.287362    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e4d5a32-ec38-4246-aaea-49002e594290-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2e4d5a32-ec38-4246-aaea-49002e594290" (UID: "2e4d5a32-ec38-4246-aaea-49002e594290"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:10:20.292665 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:20.291940    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e4d5a32-ec38-4246-aaea-49002e594290-kube-api-access-kpqtt" (OuterVolumeSpecName: "kube-api-access-kpqtt") pod "2e4d5a32-ec38-4246-aaea-49002e594290" (UID: "2e4d5a32-ec38-4246-aaea-49002e594290"). InnerVolumeSpecName "kube-api-access-kpqtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:10:20.292665 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:20.291978    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e4d5a32-ec38-4246-aaea-49002e594290-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2e4d5a32-ec38-4246-aaea-49002e594290" (UID: "2e4d5a32-ec38-4246-aaea-49002e594290"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:10:20.292665 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:20.292053    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e4d5a32-ec38-4246-aaea-49002e594290-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2e4d5a32-ec38-4246-aaea-49002e594290" (UID: "2e4d5a32-ec38-4246-aaea-49002e594290"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:10:20.387843 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:20.387805    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpqtt\" (UniqueName: \"kubernetes.io/projected/2e4d5a32-ec38-4246-aaea-49002e594290-kube-api-access-kpqtt\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:10:20.387843 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:20.387835    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e4d5a32-ec38-4246-aaea-49002e594290-console-serving-cert\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:10:20.387843 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:20.387844    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e4d5a32-ec38-4246-aaea-49002e594290-console-oauth-config\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:10:20.387843 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:20.387853    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e4d5a32-ec38-4246-aaea-49002e594290-oauth-serving-cert\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:10:20.388089 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:20.387862    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e4d5a32-ec38-4246-aaea-49002e594290-service-ca\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:10:20.388089 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:20.387870    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e4d5a32-ec38-4246-aaea-49002e594290-console-config\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:10:20.388089 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:20.387877    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e4d5a32-ec38-4246-aaea-49002e594290-trusted-ca-bundle\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:10:21.154368 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:21.154324    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-747c5cb5b4-9qkkl"
Jul 28 22:10:21.154695 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:21.154323    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-747c5cb5b4-9qkkl" event={"ID":"2e4d5a32-ec38-4246-aaea-49002e594290","Type":"ContainerDied","Data":"39812f5042818858b7543f48599879ab16a15fe261a287a0971443fbd3f97849"}
Jul 28 22:10:21.154695 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:21.154467    2563 scope.go:117] "RemoveContainer" containerID="5fed2baa11b6d2b7e602e90377c6cceb181b13494dc2eac64423e0d3600080ef"
Jul 28 22:10:21.170258 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:21.170235    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-747c5cb5b4-9qkkl"]
Jul 28 22:10:21.175419 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:21.175398    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-747c5cb5b4-9qkkl"]
Jul 28 22:10:23.128185 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:23.128145    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e4d5a32-ec38-4246-aaea-49002e594290" path="/var/lib/kubelet/pods/2e4d5a32-ec38-4246-aaea-49002e594290/volumes"
Jul 28 22:10:28.284765 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:28.284736    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ssgsl_d653889b-b9a3-4d5d-bbf0-f286413a20c9/serve-healthcheck-canary/0.log"
Jul 28 22:10:28.442627 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:28.442595    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-77cbc6f9d4-hxcn7"
Jul 28 22:10:28.446312 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:10:28.446289    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-77cbc6f9d4-hxcn7"
Jul 28 22:11:13.585681 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:13.585643    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b965f9fdc-9966c"]
Jul 28 22:11:13.586140 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:13.585882    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e4d5a32-ec38-4246-aaea-49002e594290" containerName="console"
Jul 28 22:11:13.586140 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:13.585893    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e4d5a32-ec38-4246-aaea-49002e594290" containerName="console"
Jul 28 22:11:13.586140 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:13.585939    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e4d5a32-ec38-4246-aaea-49002e594290" containerName="console"
Jul 28 22:11:13.588214 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:13.588197    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b965f9fdc-9966c"
Jul 28 22:11:13.602712 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:13.602689    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b965f9fdc-9966c"]
Jul 28 22:11:13.688822 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:13.688777    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bb2f8500-cec5-458c-8318-8e7e9bac64c4-console-oauth-config\") pod \"console-6b965f9fdc-9966c\" (UID: \"bb2f8500-cec5-458c-8318-8e7e9bac64c4\") " pod="openshift-console/console-6b965f9fdc-9966c"
Jul 28 22:11:13.688822 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:13.688824    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb2f8500-cec5-458c-8318-8e7e9bac64c4-trusted-ca-bundle\") pod \"console-6b965f9fdc-9966c\" (UID: \"bb2f8500-cec5-458c-8318-8e7e9bac64c4\") " pod="openshift-console/console-6b965f9fdc-9966c"
Jul 28 22:11:13.689036 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:13.688895    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bb2f8500-cec5-458c-8318-8e7e9bac64c4-oauth-serving-cert\") pod \"console-6b965f9fdc-9966c\" (UID: \"bb2f8500-cec5-458c-8318-8e7e9bac64c4\") " pod="openshift-console/console-6b965f9fdc-9966c"
Jul 28 22:11:13.689036 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:13.688925    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bb2f8500-cec5-458c-8318-8e7e9bac64c4-console-serving-cert\") pod \"console-6b965f9fdc-9966c\" (UID: \"bb2f8500-cec5-458c-8318-8e7e9bac64c4\") " pod="openshift-console/console-6b965f9fdc-9966c"
Jul 28 22:11:13.689036 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:13.688943    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bb2f8500-cec5-458c-8318-8e7e9bac64c4-console-config\") pod \"console-6b965f9fdc-9966c\" (UID: \"bb2f8500-cec5-458c-8318-8e7e9bac64c4\") " pod="openshift-console/console-6b965f9fdc-9966c"
Jul 28 22:11:13.689036 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:13.688965    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bb2f8500-cec5-458c-8318-8e7e9bac64c4-service-ca\") pod \"console-6b965f9fdc-9966c\" (UID: \"bb2f8500-cec5-458c-8318-8e7e9bac64c4\") " pod="openshift-console/console-6b965f9fdc-9966c"
Jul 28 22:11:13.689036 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:13.689006    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xtlg\" (UniqueName: \"kubernetes.io/projected/bb2f8500-cec5-458c-8318-8e7e9bac64c4-kube-api-access-2xtlg\") pod \"console-6b965f9fdc-9966c\" (UID: \"bb2f8500-cec5-458c-8318-8e7e9bac64c4\") " pod="openshift-console/console-6b965f9fdc-9966c"
Jul 28 22:11:13.789575 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:13.789542    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bb2f8500-cec5-458c-8318-8e7e9bac64c4-console-serving-cert\") pod \"console-6b965f9fdc-9966c\" (UID: \"bb2f8500-cec5-458c-8318-8e7e9bac64c4\") " pod="openshift-console/console-6b965f9fdc-9966c"
Jul 28 22:11:13.789742 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:13.789582    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bb2f8500-cec5-458c-8318-8e7e9bac64c4-console-config\") pod \"console-6b965f9fdc-9966c\" (UID: \"bb2f8500-cec5-458c-8318-8e7e9bac64c4\") " pod="openshift-console/console-6b965f9fdc-9966c"
Jul 28 22:11:13.789742 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:13.789611    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bb2f8500-cec5-458c-8318-8e7e9bac64c4-service-ca\") pod \"console-6b965f9fdc-9966c\" (UID: \"bb2f8500-cec5-458c-8318-8e7e9bac64c4\") " pod="openshift-console/console-6b965f9fdc-9966c"
Jul 28 22:11:13.789742 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:13.789644    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xtlg\" (UniqueName: \"kubernetes.io/projected/bb2f8500-cec5-458c-8318-8e7e9bac64c4-kube-api-access-2xtlg\") pod \"console-6b965f9fdc-9966c\" (UID: \"bb2f8500-cec5-458c-8318-8e7e9bac64c4\") " pod="openshift-console/console-6b965f9fdc-9966c"
Jul 28 22:11:13.789742 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:13.789698    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bb2f8500-cec5-458c-8318-8e7e9bac64c4-console-oauth-config\") pod \"console-6b965f9fdc-9966c\" (UID: \"bb2f8500-cec5-458c-8318-8e7e9bac64c4\") " pod="openshift-console/console-6b965f9fdc-9966c"
Jul 28 22:11:13.789742 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:13.789725    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb2f8500-cec5-458c-8318-8e7e9bac64c4-trusted-ca-bundle\") pod \"console-6b965f9fdc-9966c\" (UID: \"bb2f8500-cec5-458c-8318-8e7e9bac64c4\") " pod="openshift-console/console-6b965f9fdc-9966c"
Jul 28 22:11:13.789994 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:13.789768    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bb2f8500-cec5-458c-8318-8e7e9bac64c4-oauth-serving-cert\") pod \"console-6b965f9fdc-9966c\" (UID: \"bb2f8500-cec5-458c-8318-8e7e9bac64c4\") " pod="openshift-console/console-6b965f9fdc-9966c"
Jul 28 22:11:13.790308 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:13.790281    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bb2f8500-cec5-458c-8318-8e7e9bac64c4-service-ca\") pod \"console-6b965f9fdc-9966c\" (UID: \"bb2f8500-cec5-458c-8318-8e7e9bac64c4\") " pod="openshift-console/console-6b965f9fdc-9966c"
Jul 28 22:11:13.790308 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:13.790295    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bb2f8500-cec5-458c-8318-8e7e9bac64c4-console-config\") pod \"console-6b965f9fdc-9966c\" (UID: \"bb2f8500-cec5-458c-8318-8e7e9bac64c4\") " pod="openshift-console/console-6b965f9fdc-9966c"
Jul 28 22:11:13.790517 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:13.790482    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bb2f8500-cec5-458c-8318-8e7e9bac64c4-oauth-serving-cert\") pod \"console-6b965f9fdc-9966c\" (UID: \"bb2f8500-cec5-458c-8318-8e7e9bac64c4\") " pod="openshift-console/console-6b965f9fdc-9966c"
Jul 28 22:11:13.790601 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:13.790582    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb2f8500-cec5-458c-8318-8e7e9bac64c4-trusted-ca-bundle\") pod \"console-6b965f9fdc-9966c\" (UID: \"bb2f8500-cec5-458c-8318-8e7e9bac64c4\") " pod="openshift-console/console-6b965f9fdc-9966c"
Jul 28 22:11:13.792161 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:13.792131    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bb2f8500-cec5-458c-8318-8e7e9bac64c4-console-oauth-config\") pod \"console-6b965f9fdc-9966c\" (UID: \"bb2f8500-cec5-458c-8318-8e7e9bac64c4\") " pod="openshift-console/console-6b965f9fdc-9966c"
Jul 28 22:11:13.792245 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:13.792138    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bb2f8500-cec5-458c-8318-8e7e9bac64c4-console-serving-cert\") pod \"console-6b965f9fdc-9966c\" (UID: \"bb2f8500-cec5-458c-8318-8e7e9bac64c4\") " pod="openshift-console/console-6b965f9fdc-9966c"
Jul 28 22:11:13.798058 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:13.798037    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xtlg\" (UniqueName: \"kubernetes.io/projected/bb2f8500-cec5-458c-8318-8e7e9bac64c4-kube-api-access-2xtlg\") pod \"console-6b965f9fdc-9966c\" (UID: \"bb2f8500-cec5-458c-8318-8e7e9bac64c4\") " pod="openshift-console/console-6b965f9fdc-9966c"
Jul 28 22:11:13.897099 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:13.897025    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b965f9fdc-9966c"
Jul 28 22:11:14.021412 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:14.021377    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b965f9fdc-9966c"]
Jul 28 22:11:14.023653 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:11:14.023627    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2f8500_cec5_458c_8318_8e7e9bac64c4.slice/crio-6788665e21cdc5ba32269bae97fa091974597513fe51fd146e6d9ce273f843f1 WatchSource:0}: Error finding container 6788665e21cdc5ba32269bae97fa091974597513fe51fd146e6d9ce273f843f1: Status 404 returned error can't find the container with id 6788665e21cdc5ba32269bae97fa091974597513fe51fd146e6d9ce273f843f1
Jul 28 22:11:14.328123 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:14.328092    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b965f9fdc-9966c" event={"ID":"bb2f8500-cec5-458c-8318-8e7e9bac64c4","Type":"ContainerStarted","Data":"b2b57437c359056a9ddcd4567e1532bc2a18725d27b835f28f013dd9204bf22b"}
Jul 28 22:11:14.328303 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:14.328145    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b965f9fdc-9966c" event={"ID":"bb2f8500-cec5-458c-8318-8e7e9bac64c4","Type":"ContainerStarted","Data":"6788665e21cdc5ba32269bae97fa091974597513fe51fd146e6d9ce273f843f1"}
Jul 28 22:11:14.346440 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:14.346398    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b965f9fdc-9966c" podStartSLOduration=1.346383198 podStartE2EDuration="1.346383198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:11:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:11:14.345325659 +0000 UTC m=+371.725463305" watchObservedRunningTime="2026-07-28 22:11:14.346383198 +0000 UTC m=+371.726520838"
Jul 28 22:11:17.019210 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:17.019176    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f33e531-3341-4c01-940e-91f17519976c-metrics-certs\") pod \"network-metrics-daemon-4xqhf\" (UID: \"2f33e531-3341-4c01-940e-91f17519976c\") " pod="openshift-multus/network-metrics-daemon-4xqhf"
Jul 28 22:11:17.021490 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:17.021469    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f33e531-3341-4c01-940e-91f17519976c-metrics-certs\") pod \"network-metrics-daemon-4xqhf\" (UID: \"2f33e531-3341-4c01-940e-91f17519976c\") " pod="openshift-multus/network-metrics-daemon-4xqhf"
Jul 28 22:11:17.027820 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:17.027797    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f5pt7\""
Jul 28 22:11:17.035765 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:17.035751    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4xqhf"
Jul 28 22:11:17.182000 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:17.181966    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4xqhf"]
Jul 28 22:11:17.185391 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:11:17.185344    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f33e531_3341_4c01_940e_91f17519976c.slice/crio-59241b38e036e60f36453e595a64b6bea8c3e1039b67112dc52c58c726e62841 WatchSource:0}: Error finding container 59241b38e036e60f36453e595a64b6bea8c3e1039b67112dc52c58c726e62841: Status 404 returned error can't find the container with id 59241b38e036e60f36453e595a64b6bea8c3e1039b67112dc52c58c726e62841
Jul 28 22:11:17.187408 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:17.187390    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 22:11:17.339686 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:17.339608    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4xqhf" event={"ID":"2f33e531-3341-4c01-940e-91f17519976c","Type":"ContainerStarted","Data":"59241b38e036e60f36453e595a64b6bea8c3e1039b67112dc52c58c726e62841"}
Jul 28 22:11:18.348671 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:18.348582    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4xqhf" event={"ID":"2f33e531-3341-4c01-940e-91f17519976c","Type":"ContainerStarted","Data":"07f48f505c254a1b66825df997c38f61d0fbf9d1543a70d86d8be4356a354bff"}
Jul 28 22:11:18.348671 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:18.348660    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4xqhf" event={"ID":"2f33e531-3341-4c01-940e-91f17519976c","Type":"ContainerStarted","Data":"d6fecd0ffc418fbcaa0078b1769e95064242a9c05d27c2e9a859e2c52a715131"}
Jul 28 22:11:18.364519 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:18.364474    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4xqhf" podStartSLOduration=373.486784875 podStartE2EDuration="6m14.364459854s" totalImagesPullingTime="877.674979ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:05:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:11:18.363759817 +0000 UTC m=+375.743897462" watchObservedRunningTime="2026-07-28 22:11:18.364459854 +0000 UTC m=+375.744597498"
Jul 28 22:11:23.898207 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:23.898167    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b965f9fdc-9966c"
Jul 28 22:11:23.898639 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:23.898257    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b965f9fdc-9966c"
Jul 28 22:11:23.903139 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:23.903114    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b965f9fdc-9966c"
Jul 28 22:11:24.374343 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:24.374310    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b965f9fdc-9966c"
Jul 28 22:11:24.421684 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:24.421643    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c57655f7f-nsfd4"]
Jul 28 22:11:24.503936 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:24.503902    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-dd875b845-229jm"]
Jul 28 22:11:24.506043 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:24.506027    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dd875b845-229jm"
Jul 28 22:11:24.518184 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:24.518161    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dd875b845-229jm"]
Jul 28 22:11:24.573814 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:24.573787    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3757325f-8310-4c4d-bf0a-8686acf3c88c-trusted-ca-bundle\") pod \"console-dd875b845-229jm\" (UID: \"3757325f-8310-4c4d-bf0a-8686acf3c88c\") " pod="openshift-console/console-dd875b845-229jm"
Jul 28 22:11:24.573814 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:24.573819    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3757325f-8310-4c4d-bf0a-8686acf3c88c-console-serving-cert\") pod \"console-dd875b845-229jm\" (UID: \"3757325f-8310-4c4d-bf0a-8686acf3c88c\") " pod="openshift-console/console-dd875b845-229jm"
Jul 28 22:11:24.574038 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:24.573857    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3757325f-8310-4c4d-bf0a-8686acf3c88c-console-oauth-config\") pod \"console-dd875b845-229jm\" (UID: \"3757325f-8310-4c4d-bf0a-8686acf3c88c\") " pod="openshift-console/console-dd875b845-229jm"
Jul 28 22:11:24.574038 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:24.573900    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3757325f-8310-4c4d-bf0a-8686acf3c88c-console-config\") pod \"console-dd875b845-229jm\" (UID: \"3757325f-8310-4c4d-bf0a-8686acf3c88c\") " pod="openshift-console/console-dd875b845-229jm"
Jul 28 22:11:24.574038 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:24.573952    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3757325f-8310-4c4d-bf0a-8686acf3c88c-service-ca\") pod \"console-dd875b845-229jm\" (UID: \"3757325f-8310-4c4d-bf0a-8686acf3c88c\") " pod="openshift-console/console-dd875b845-229jm"
Jul 28 22:11:24.574038 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:24.573993    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3757325f-8310-4c4d-bf0a-8686acf3c88c-oauth-serving-cert\") pod \"console-dd875b845-229jm\" (UID: \"3757325f-8310-4c4d-bf0a-8686acf3c88c\") " pod="openshift-console/console-dd875b845-229jm"
Jul 28 22:11:24.574038 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:24.574031    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l84sk\" (UniqueName: \"kubernetes.io/projected/3757325f-8310-4c4d-bf0a-8686acf3c88c-kube-api-access-l84sk\") pod \"console-dd875b845-229jm\" (UID: \"3757325f-8310-4c4d-bf0a-8686acf3c88c\") " pod="openshift-console/console-dd875b845-229jm"
Jul 28 22:11:24.674844 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:24.674762    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3757325f-8310-4c4d-bf0a-8686acf3c88c-console-oauth-config\") pod \"console-dd875b845-229jm\" (UID: \"3757325f-8310-4c4d-bf0a-8686acf3c88c\") " pod="openshift-console/console-dd875b845-229jm"
Jul 28 22:11:24.674844 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:24.674798    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3757325f-8310-4c4d-bf0a-8686acf3c88c-console-config\") pod \"console-dd875b845-229jm\" (UID: \"3757325f-8310-4c4d-bf0a-8686acf3c88c\") " pod="openshift-console/console-dd875b845-229jm"
Jul 28 22:11:24.674844 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:24.674830    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3757325f-8310-4c4d-bf0a-8686acf3c88c-service-ca\") pod \"console-dd875b845-229jm\" (UID: \"3757325f-8310-4c4d-bf0a-8686acf3c88c\") " pod="openshift-console/console-dd875b845-229jm"
Jul 28 22:11:24.675078 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:24.674874    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3757325f-8310-4c4d-bf0a-8686acf3c88c-oauth-serving-cert\") pod \"console-dd875b845-229jm\" (UID: \"3757325f-8310-4c4d-bf0a-8686acf3c88c\") " pod="openshift-console/console-dd875b845-229jm"
Jul 28 22:11:24.675078 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:24.674909    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l84sk\" (UniqueName: \"kubernetes.io/projected/3757325f-8310-4c4d-bf0a-8686acf3c88c-kube-api-access-l84sk\") pod \"console-dd875b845-229jm\" (UID: \"3757325f-8310-4c4d-bf0a-8686acf3c88c\") " pod="openshift-console/console-dd875b845-229jm"
Jul 28 22:11:24.675078 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:24.674939    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3757325f-8310-4c4d-bf0a-8686acf3c88c-trusted-ca-bundle\") pod \"console-dd875b845-229jm\" (UID: \"3757325f-8310-4c4d-bf0a-8686acf3c88c\") " pod="openshift-console/console-dd875b845-229jm"
Jul 28 22:11:24.675078 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:24.674963    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3757325f-8310-4c4d-bf0a-8686acf3c88c-console-serving-cert\") pod \"console-dd875b845-229jm\" (UID: \"3757325f-8310-4c4d-bf0a-8686acf3c88c\") " pod="openshift-console/console-dd875b845-229jm"
Jul 28 22:11:24.675618 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:24.675593    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3757325f-8310-4c4d-bf0a-8686acf3c88c-console-config\") pod \"console-dd875b845-229jm\" (UID: \"3757325f-8310-4c4d-bf0a-8686acf3c88c\") " pod="openshift-console/console-dd875b845-229jm"
Jul 28 22:11:24.675717 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:24.675692    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3757325f-8310-4c4d-bf0a-8686acf3c88c-service-ca\") pod \"console-dd875b845-229jm\" (UID: \"3757325f-8310-4c4d-bf0a-8686acf3c88c\") " pod="openshift-console/console-dd875b845-229jm"
Jul 28 22:11:24.675794 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:24.675749    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3757325f-8310-4c4d-bf0a-8686acf3c88c-oauth-serving-cert\") pod \"console-dd875b845-229jm\" (UID: \"3757325f-8310-4c4d-bf0a-8686acf3c88c\") " pod="openshift-console/console-dd875b845-229jm"
Jul 28 22:11:24.675853 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:24.675830    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3757325f-8310-4c4d-bf0a-8686acf3c88c-trusted-ca-bundle\") pod \"console-dd875b845-229jm\" (UID: \"3757325f-8310-4c4d-bf0a-8686acf3c88c\") " pod="openshift-console/console-dd875b845-229jm"
Jul 28 22:11:24.677331 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:24.677303    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3757325f-8310-4c4d-bf0a-8686acf3c88c-console-oauth-config\") pod \"console-dd875b845-229jm\" (UID: \"3757325f-8310-4c4d-bf0a-8686acf3c88c\") " pod="openshift-console/console-dd875b845-229jm"
Jul 28 22:11:24.677548 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:24.677528    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3757325f-8310-4c4d-bf0a-8686acf3c88c-console-serving-cert\") pod \"console-dd875b845-229jm\" (UID: \"3757325f-8310-4c4d-bf0a-8686acf3c88c\") " pod="openshift-console/console-dd875b845-229jm"
Jul 28 22:11:24.689753 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:24.689729    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l84sk\" (UniqueName: \"kubernetes.io/projected/3757325f-8310-4c4d-bf0a-8686acf3c88c-kube-api-access-l84sk\") pod \"console-dd875b845-229jm\" (UID: \"3757325f-8310-4c4d-bf0a-8686acf3c88c\") " pod="openshift-console/console-dd875b845-229jm"
Jul 28 22:11:24.814712 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:24.814682    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dd875b845-229jm"
Jul 28 22:11:24.939097 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:24.938960    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dd875b845-229jm"]
Jul 28 22:11:24.941723 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:11:24.941695    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3757325f_8310_4c4d_bf0a_8686acf3c88c.slice/crio-49eaa140a15e344650c796a77f1f24cc7e7a7f202eefc4ec5b42a6275e3561e6 WatchSource:0}: Error finding container 49eaa140a15e344650c796a77f1f24cc7e7a7f202eefc4ec5b42a6275e3561e6: Status 404 returned error can't find the container with id 49eaa140a15e344650c796a77f1f24cc7e7a7f202eefc4ec5b42a6275e3561e6
Jul 28 22:11:25.376003 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:25.375908    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dd875b845-229jm" event={"ID":"3757325f-8310-4c4d-bf0a-8686acf3c88c","Type":"ContainerStarted","Data":"96a3e22c1c4b0ea23a7b761d3608129466cc3f50b93f4c1956b3ad72161095d8"}
Jul 28 22:11:25.376003 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:25.375960    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dd875b845-229jm" event={"ID":"3757325f-8310-4c4d-bf0a-8686acf3c88c","Type":"ContainerStarted","Data":"49eaa140a15e344650c796a77f1f24cc7e7a7f202eefc4ec5b42a6275e3561e6"}
Jul 28 22:11:25.406066 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:25.406021    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-dd875b845-229jm" podStartSLOduration=1.406002759 podStartE2EDuration="1.406002759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:11:25.404668979 +0000 UTC m=+382.784806623" watchObservedRunningTime="2026-07-28 22:11:25.406002759 +0000 UTC m=+382.786140407"
Jul 28 22:11:34.815743 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:34.815708    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-dd875b845-229jm"
Jul 28 22:11:34.816182 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:34.815791    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-dd875b845-229jm"
Jul 28 22:11:34.820503 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:34.820480    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-dd875b845-229jm"
Jul 28 22:11:35.414768 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:35.414739    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-dd875b845-229jm"
Jul 28 22:11:35.496248 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:35.496211    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b965f9fdc-9966c"]
Jul 28 22:11:49.441871 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:49.441826    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-c57655f7f-nsfd4" podUID="fd8a3086-bf14-4ad1-bf98-f8fa765aceab" containerName="console" containerID="cri-o://21d9e8cff459a20c5e035c882b9b47098f67af7cbc07a52209b60c174270fabc" gracePeriod=15
Jul 28 22:11:49.684973 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:49.684941    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c57655f7f-nsfd4"
Jul 28 22:11:49.776753 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:49.776682    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-console-oauth-config\") pod \"fd8a3086-bf14-4ad1-bf98-f8fa765aceab\" (UID: \"fd8a3086-bf14-4ad1-bf98-f8fa765aceab\") "
Jul 28 22:11:49.776753 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:49.776717    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-console-serving-cert\") pod \"fd8a3086-bf14-4ad1-bf98-f8fa765aceab\" (UID: \"fd8a3086-bf14-4ad1-bf98-f8fa765aceab\") "
Jul 28 22:11:49.776753 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:49.776748    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-trusted-ca-bundle\") pod \"fd8a3086-bf14-4ad1-bf98-f8fa765aceab\" (UID: \"fd8a3086-bf14-4ad1-bf98-f8fa765aceab\") "
Jul 28 22:11:49.777037 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:49.776796    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8p2xv\" (UniqueName: \"kubernetes.io/projected/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-kube-api-access-8p2xv\") pod \"fd8a3086-bf14-4ad1-bf98-f8fa765aceab\" (UID: \"fd8a3086-bf14-4ad1-bf98-f8fa765aceab\") "
Jul 28 22:11:49.777037 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:49.776832    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-console-config\") pod \"fd8a3086-bf14-4ad1-bf98-f8fa765aceab\" (UID: \"fd8a3086-bf14-4ad1-bf98-f8fa765aceab\") "
Jul 28 22:11:49.777037 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:49.776894    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-service-ca\") pod \"fd8a3086-bf14-4ad1-bf98-f8fa765aceab\" (UID: \"fd8a3086-bf14-4ad1-bf98-f8fa765aceab\") "
Jul 28 22:11:49.777037 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:49.776922    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-oauth-serving-cert\") pod \"fd8a3086-bf14-4ad1-bf98-f8fa765aceab\" (UID: \"fd8a3086-bf14-4ad1-bf98-f8fa765aceab\") "
Jul 28 22:11:49.777394 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:49.777336    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-console-config" (OuterVolumeSpecName: "console-config") pod "fd8a3086-bf14-4ad1-bf98-f8fa765aceab" (UID: "fd8a3086-bf14-4ad1-bf98-f8fa765aceab"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:11:49.777394 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:49.777330    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-service-ca" (OuterVolumeSpecName: "service-ca") pod "fd8a3086-bf14-4ad1-bf98-f8fa765aceab" (UID: "fd8a3086-bf14-4ad1-bf98-f8fa765aceab"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:11:49.777394 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:49.777382    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "fd8a3086-bf14-4ad1-bf98-f8fa765aceab" (UID: "fd8a3086-bf14-4ad1-bf98-f8fa765aceab"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:11:49.777574 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:49.777397    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fd8a3086-bf14-4ad1-bf98-f8fa765aceab" (UID: "fd8a3086-bf14-4ad1-bf98-f8fa765aceab"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:11:49.777621 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:49.777604    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-trusted-ca-bundle\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:11:49.777656 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:49.777629    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-console-config\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:11:49.777656 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:49.777642    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-service-ca\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:11:49.777715 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:49.777655    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-oauth-serving-cert\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:11:49.778873 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:49.778844    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fd8a3086-bf14-4ad1-bf98-f8fa765aceab" (UID: "fd8a3086-bf14-4ad1-bf98-f8fa765aceab"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:11:49.778976 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:49.778889    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-kube-api-access-8p2xv" (OuterVolumeSpecName: "kube-api-access-8p2xv") pod "fd8a3086-bf14-4ad1-bf98-f8fa765aceab" (UID: "fd8a3086-bf14-4ad1-bf98-f8fa765aceab"). InnerVolumeSpecName "kube-api-access-8p2xv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:11:49.778976 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:49.778912    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fd8a3086-bf14-4ad1-bf98-f8fa765aceab" (UID: "fd8a3086-bf14-4ad1-bf98-f8fa765aceab"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:11:49.878444 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:49.878418    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-console-oauth-config\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:11:49.878444 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:49.878441    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-console-serving-cert\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:11:49.878444 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:49.878450    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p2xv\" (UniqueName: \"kubernetes.io/projected/fd8a3086-bf14-4ad1-bf98-f8fa765aceab-kube-api-access-8p2xv\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:11:50.462060 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:50.462020    2563 generic.go:356] "Generic (PLEG): container finished" podID="fd8a3086-bf14-4ad1-bf98-f8fa765aceab" containerID="21d9e8cff459a20c5e035c882b9b47098f67af7cbc07a52209b60c174270fabc" exitCode=0
Jul 28 22:11:50.462485 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:50.462065    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c57655f7f-nsfd4"
Jul 28 22:11:50.462485 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:50.462104    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c57655f7f-nsfd4" event={"ID":"fd8a3086-bf14-4ad1-bf98-f8fa765aceab","Type":"ContainerDied","Data":"21d9e8cff459a20c5e035c882b9b47098f67af7cbc07a52209b60c174270fabc"}
Jul 28 22:11:50.462485 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:50.462157    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c57655f7f-nsfd4" event={"ID":"fd8a3086-bf14-4ad1-bf98-f8fa765aceab","Type":"ContainerDied","Data":"640454a87b2cd2a27b489f144d1a0685467e1247d69a2ebe5e37ba4ad1eddd06"}
Jul 28 22:11:50.462485 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:50.462178    2563 scope.go:117] "RemoveContainer" containerID="21d9e8cff459a20c5e035c882b9b47098f67af7cbc07a52209b60c174270fabc"
Jul 28 22:11:50.470375 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:50.470339    2563 scope.go:117] "RemoveContainer" containerID="21d9e8cff459a20c5e035c882b9b47098f67af7cbc07a52209b60c174270fabc"
Jul 28 22:11:50.470602 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:11:50.470585    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21d9e8cff459a20c5e035c882b9b47098f67af7cbc07a52209b60c174270fabc\": container with ID starting with 21d9e8cff459a20c5e035c882b9b47098f67af7cbc07a52209b60c174270fabc not found: ID does not exist" containerID="21d9e8cff459a20c5e035c882b9b47098f67af7cbc07a52209b60c174270fabc"
Jul 28 22:11:50.470656 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:50.470610    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21d9e8cff459a20c5e035c882b9b47098f67af7cbc07a52209b60c174270fabc"} err="failed to get container status \"21d9e8cff459a20c5e035c882b9b47098f67af7cbc07a52209b60c174270fabc\": rpc error: code = NotFound desc = could not find container \"21d9e8cff459a20c5e035c882b9b47098f67af7cbc07a52209b60c174270fabc\": container with ID starting with 21d9e8cff459a20c5e035c882b9b47098f67af7cbc07a52209b60c174270fabc not found: ID does not exist"
Jul 28 22:11:50.484845 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:50.484819    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c57655f7f-nsfd4"]
Jul 28 22:11:50.489209 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:50.489189    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c57655f7f-nsfd4"]
Jul 28 22:11:51.128542 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:11:51.128510    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd8a3086-bf14-4ad1-bf98-f8fa765aceab" path="/var/lib/kubelet/pods/fd8a3086-bf14-4ad1-bf98-f8fa765aceab/volumes"
Jul 28 22:12:00.515700 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:00.515647    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6b965f9fdc-9966c" podUID="bb2f8500-cec5-458c-8318-8e7e9bac64c4" containerName="console" containerID="cri-o://b2b57437c359056a9ddcd4567e1532bc2a18725d27b835f28f013dd9204bf22b" gracePeriod=15
Jul 28 22:12:00.751532 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:00.751505    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b965f9fdc-9966c"
Jul 28 22:12:00.865801 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:00.865710    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bb2f8500-cec5-458c-8318-8e7e9bac64c4-oauth-serving-cert\") pod \"bb2f8500-cec5-458c-8318-8e7e9bac64c4\" (UID: \"bb2f8500-cec5-458c-8318-8e7e9bac64c4\") "
Jul 28 22:12:00.865801 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:00.865742    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bb2f8500-cec5-458c-8318-8e7e9bac64c4-service-ca\") pod \"bb2f8500-cec5-458c-8318-8e7e9bac64c4\" (UID: \"bb2f8500-cec5-458c-8318-8e7e9bac64c4\") "
Jul 28 22:12:00.865801 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:00.865769    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb2f8500-cec5-458c-8318-8e7e9bac64c4-trusted-ca-bundle\") pod \"bb2f8500-cec5-458c-8318-8e7e9bac64c4\" (UID: \"bb2f8500-cec5-458c-8318-8e7e9bac64c4\") "
Jul 28 22:12:00.866080 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:00.865813    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bb2f8500-cec5-458c-8318-8e7e9bac64c4-console-config\") pod \"bb2f8500-cec5-458c-8318-8e7e9bac64c4\" (UID: \"bb2f8500-cec5-458c-8318-8e7e9bac64c4\") "
Jul 28 22:12:00.866080 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:00.865864    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xtlg\" (UniqueName: \"kubernetes.io/projected/bb2f8500-cec5-458c-8318-8e7e9bac64c4-kube-api-access-2xtlg\") pod \"bb2f8500-cec5-458c-8318-8e7e9bac64c4\" (UID: \"bb2f8500-cec5-458c-8318-8e7e9bac64c4\") "
Jul 28 22:12:00.866080 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:00.865921    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bb2f8500-cec5-458c-8318-8e7e9bac64c4-console-oauth-config\") pod \"bb2f8500-cec5-458c-8318-8e7e9bac64c4\" (UID: \"bb2f8500-cec5-458c-8318-8e7e9bac64c4\") "
Jul 28 22:12:00.866080 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:00.865950    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bb2f8500-cec5-458c-8318-8e7e9bac64c4-console-serving-cert\") pod \"bb2f8500-cec5-458c-8318-8e7e9bac64c4\" (UID: \"bb2f8500-cec5-458c-8318-8e7e9bac64c4\") "
Jul 28 22:12:00.866271 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:00.866123    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb2f8500-cec5-458c-8318-8e7e9bac64c4-service-ca" (OuterVolumeSpecName: "service-ca") pod "bb2f8500-cec5-458c-8318-8e7e9bac64c4" (UID: "bb2f8500-cec5-458c-8318-8e7e9bac64c4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:12:00.866271 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:00.866227    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb2f8500-cec5-458c-8318-8e7e9bac64c4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "bb2f8500-cec5-458c-8318-8e7e9bac64c4" (UID: "bb2f8500-cec5-458c-8318-8e7e9bac64c4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:12:00.866387 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:00.866330    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb2f8500-cec5-458c-8318-8e7e9bac64c4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bb2f8500-cec5-458c-8318-8e7e9bac64c4" (UID: "bb2f8500-cec5-458c-8318-8e7e9bac64c4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:12:00.866387 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:00.866373    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb2f8500-cec5-458c-8318-8e7e9bac64c4-console-config" (OuterVolumeSpecName: "console-config") pod "bb2f8500-cec5-458c-8318-8e7e9bac64c4" (UID: "bb2f8500-cec5-458c-8318-8e7e9bac64c4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:12:00.868142 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:00.868109    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb2f8500-cec5-458c-8318-8e7e9bac64c4-kube-api-access-2xtlg" (OuterVolumeSpecName: "kube-api-access-2xtlg") pod "bb2f8500-cec5-458c-8318-8e7e9bac64c4" (UID: "bb2f8500-cec5-458c-8318-8e7e9bac64c4"). InnerVolumeSpecName "kube-api-access-2xtlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:12:00.868142 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:00.868126    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb2f8500-cec5-458c-8318-8e7e9bac64c4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bb2f8500-cec5-458c-8318-8e7e9bac64c4" (UID: "bb2f8500-cec5-458c-8318-8e7e9bac64c4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:12:00.868301 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:00.868151    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb2f8500-cec5-458c-8318-8e7e9bac64c4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bb2f8500-cec5-458c-8318-8e7e9bac64c4" (UID: "bb2f8500-cec5-458c-8318-8e7e9bac64c4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:12:00.966760 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:00.966721    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xtlg\" (UniqueName: \"kubernetes.io/projected/bb2f8500-cec5-458c-8318-8e7e9bac64c4-kube-api-access-2xtlg\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:12:00.966760 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:00.966752    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bb2f8500-cec5-458c-8318-8e7e9bac64c4-console-oauth-config\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:12:00.966760 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:00.966762    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bb2f8500-cec5-458c-8318-8e7e9bac64c4-console-serving-cert\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:12:00.966760 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:00.966771    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bb2f8500-cec5-458c-8318-8e7e9bac64c4-oauth-serving-cert\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:12:00.967013 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:00.966781    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bb2f8500-cec5-458c-8318-8e7e9bac64c4-service-ca\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:12:00.967013 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:00.966789    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb2f8500-cec5-458c-8318-8e7e9bac64c4-trusted-ca-bundle\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:12:00.967013 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:00.966797    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bb2f8500-cec5-458c-8318-8e7e9bac64c4-console-config\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:12:01.500629 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:01.500597    2563 generic.go:356] "Generic (PLEG): container finished" podID="bb2f8500-cec5-458c-8318-8e7e9bac64c4" containerID="b2b57437c359056a9ddcd4567e1532bc2a18725d27b835f28f013dd9204bf22b" exitCode=0
Jul 28 22:12:01.500789 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:01.500640    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b965f9fdc-9966c"
Jul 28 22:12:01.500789 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:01.500684    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b965f9fdc-9966c" event={"ID":"bb2f8500-cec5-458c-8318-8e7e9bac64c4","Type":"ContainerDied","Data":"b2b57437c359056a9ddcd4567e1532bc2a18725d27b835f28f013dd9204bf22b"}
Jul 28 22:12:01.500789 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:01.500736    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b965f9fdc-9966c" event={"ID":"bb2f8500-cec5-458c-8318-8e7e9bac64c4","Type":"ContainerDied","Data":"6788665e21cdc5ba32269bae97fa091974597513fe51fd146e6d9ce273f843f1"}
Jul 28 22:12:01.500789 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:01.500767    2563 scope.go:117] "RemoveContainer" containerID="b2b57437c359056a9ddcd4567e1532bc2a18725d27b835f28f013dd9204bf22b"
Jul 28 22:12:01.508400 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:01.508383    2563 scope.go:117] "RemoveContainer" containerID="b2b57437c359056a9ddcd4567e1532bc2a18725d27b835f28f013dd9204bf22b"
Jul 28 22:12:01.508648 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:12:01.508630    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2b57437c359056a9ddcd4567e1532bc2a18725d27b835f28f013dd9204bf22b\": container with ID starting with b2b57437c359056a9ddcd4567e1532bc2a18725d27b835f28f013dd9204bf22b not found: ID does not exist" containerID="b2b57437c359056a9ddcd4567e1532bc2a18725d27b835f28f013dd9204bf22b"
Jul 28 22:12:01.508716 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:01.508663    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2b57437c359056a9ddcd4567e1532bc2a18725d27b835f28f013dd9204bf22b"} err="failed to get container status \"b2b57437c359056a9ddcd4567e1532bc2a18725d27b835f28f013dd9204bf22b\": rpc error: code = NotFound desc = could not find container \"b2b57437c359056a9ddcd4567e1532bc2a18725d27b835f28f013dd9204bf22b\": container with ID starting with b2b57437c359056a9ddcd4567e1532bc2a18725d27b835f28f013dd9204bf22b not found: ID does not exist"
Jul 28 22:12:01.530698 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:01.530669    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b965f9fdc-9966c"]
Jul 28 22:12:01.547089 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:01.547066    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b965f9fdc-9966c"]
Jul 28 22:12:03.129510 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:03.129480    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb2f8500-cec5-458c-8318-8e7e9bac64c4" path="/var/lib/kubelet/pods/bb2f8500-cec5-458c-8318-8e7e9bac64c4/volumes"
Jul 28 22:12:37.343280 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:37.343246    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-ss7fs"]
Jul 28 22:12:37.343773 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:37.343509    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb2f8500-cec5-458c-8318-8e7e9bac64c4" containerName="console"
Jul 28 22:12:37.343773 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:37.343521    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb2f8500-cec5-458c-8318-8e7e9bac64c4" containerName="console"
Jul 28 22:12:37.343773 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:37.343533    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd8a3086-bf14-4ad1-bf98-f8fa765aceab" containerName="console"
Jul 28 22:12:37.343773 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:37.343538    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd8a3086-bf14-4ad1-bf98-f8fa765aceab" containerName="console"
Jul 28 22:12:37.343773 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:37.343578    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd8a3086-bf14-4ad1-bf98-f8fa765aceab" containerName="console"
Jul 28 22:12:37.343773 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:37.343587    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb2f8500-cec5-458c-8318-8e7e9bac64c4" containerName="console"
Jul 28 22:12:37.346203 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:37.346186    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-ss7fs"
Jul 28 22:12:37.349520 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:37.349499    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 28 22:12:37.350376 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:37.350346    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-lsjdn\""
Jul 28 22:12:37.350470 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:37.350364    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 28 22:12:37.374378 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:37.372455    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-ss7fs"]
Jul 28 22:12:37.429645 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:37.429613    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6416ea94-b746-48b3-8750-c2667c3c5b84-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-ss7fs\" (UID: \"6416ea94-b746-48b3-8750-c2667c3c5b84\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ss7fs"
Jul 28 22:12:37.429771 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:37.429651    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4ckj\" (UniqueName: \"kubernetes.io/projected/6416ea94-b746-48b3-8750-c2667c3c5b84-kube-api-access-c4ckj\") pod \"cert-manager-webhook-6486ffd997-ss7fs\" (UID: \"6416ea94-b746-48b3-8750-c2667c3c5b84\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ss7fs"
Jul 28 22:12:37.530749 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:37.530720    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6416ea94-b746-48b3-8750-c2667c3c5b84-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-ss7fs\" (UID: \"6416ea94-b746-48b3-8750-c2667c3c5b84\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ss7fs"
Jul 28 22:12:37.530889 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:37.530760    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4ckj\" (UniqueName: \"kubernetes.io/projected/6416ea94-b746-48b3-8750-c2667c3c5b84-kube-api-access-c4ckj\") pod \"cert-manager-webhook-6486ffd997-ss7fs\" (UID: \"6416ea94-b746-48b3-8750-c2667c3c5b84\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ss7fs"
Jul 28 22:12:37.539367 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:37.539324    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6416ea94-b746-48b3-8750-c2667c3c5b84-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-ss7fs\" (UID: \"6416ea94-b746-48b3-8750-c2667c3c5b84\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ss7fs"
Jul 28 22:12:37.539846 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:37.539827    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4ckj\" (UniqueName: \"kubernetes.io/projected/6416ea94-b746-48b3-8750-c2667c3c5b84-kube-api-access-c4ckj\") pod \"cert-manager-webhook-6486ffd997-ss7fs\" (UID: \"6416ea94-b746-48b3-8750-c2667c3c5b84\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ss7fs"
Jul 28 22:12:37.654660 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:37.654586    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-ss7fs"
Jul 28 22:12:37.779843 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:37.779770    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-ss7fs"]
Jul 28 22:12:37.781768 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:12:37.781738    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6416ea94_b746_48b3_8750_c2667c3c5b84.slice/crio-f5eb2461e15f3f8ac585cd0f703cf08a6c09078d9434957d0e58eed50b9e178b WatchSource:0}: Error finding container f5eb2461e15f3f8ac585cd0f703cf08a6c09078d9434957d0e58eed50b9e178b: Status 404 returned error can't find the container with id f5eb2461e15f3f8ac585cd0f703cf08a6c09078d9434957d0e58eed50b9e178b
Jul 28 22:12:38.632562 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:38.632523    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-ss7fs" event={"ID":"6416ea94-b746-48b3-8750-c2667c3c5b84","Type":"ContainerStarted","Data":"f5eb2461e15f3f8ac585cd0f703cf08a6c09078d9434957d0e58eed50b9e178b"}
Jul 28 22:12:41.644624 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:41.644587    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-ss7fs" event={"ID":"6416ea94-b746-48b3-8750-c2667c3c5b84","Type":"ContainerStarted","Data":"f23c8dfffbeab7ea930627cac9de2676ccbb1c4fa40876ebdc2c9eebdf80b4d0"}
Jul 28 22:12:41.645102 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:41.644648    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-ss7fs"
Jul 28 22:12:41.660937 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:41.660891    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-ss7fs" podStartSLOduration=1.6541151360000002 podStartE2EDuration="4.660875199s" totalImagesPullingTime="3.006760063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:12:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:12:41.65936945 +0000 UTC m=+459.039507084" watchObservedRunningTime="2026-07-28 22:12:41.660875199 +0000 UTC m=+459.041012843"
Jul 28 22:12:47.650969 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:47.650938    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-ss7fs"
Jul 28 22:12:51.563605 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:51.563571    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pmlf6"]
Jul 28 22:12:51.566646 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:51.566630    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pmlf6"
Jul 28 22:12:51.569185 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:51.569158    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 28 22:12:51.569476 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:51.569459    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 28 22:12:51.569550 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:51.569526    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-k2fzt\""
Jul 28 22:12:51.575902 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:51.575882    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pmlf6"]
Jul 28 22:12:51.639288 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:51.639256    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xb8c\" (UniqueName: \"kubernetes.io/projected/f88674d2-586a-4fd9-ac95-711f572c5dc1-kube-api-access-2xb8c\") pod \"openshift-lws-operator-fd8ccff4c-pmlf6\" (UID: \"f88674d2-586a-4fd9-ac95-711f572c5dc1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pmlf6"
Jul 28 22:12:51.639453 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:51.639309    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f88674d2-586a-4fd9-ac95-711f572c5dc1-tmp\") pod \"openshift-lws-operator-fd8ccff4c-pmlf6\" (UID: \"f88674d2-586a-4fd9-ac95-711f572c5dc1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pmlf6"
Jul 28 22:12:51.739986 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:51.739948    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xb8c\" (UniqueName: \"kubernetes.io/projected/f88674d2-586a-4fd9-ac95-711f572c5dc1-kube-api-access-2xb8c\") pod \"openshift-lws-operator-fd8ccff4c-pmlf6\" (UID: \"f88674d2-586a-4fd9-ac95-711f572c5dc1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pmlf6"
Jul 28 22:12:51.740141 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:51.739998    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f88674d2-586a-4fd9-ac95-711f572c5dc1-tmp\") pod \"openshift-lws-operator-fd8ccff4c-pmlf6\" (UID: \"f88674d2-586a-4fd9-ac95-711f572c5dc1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pmlf6"
Jul 28 22:12:51.740329 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:51.740314    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f88674d2-586a-4fd9-ac95-711f572c5dc1-tmp\") pod \"openshift-lws-operator-fd8ccff4c-pmlf6\" (UID: \"f88674d2-586a-4fd9-ac95-711f572c5dc1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pmlf6"
Jul 28 22:12:51.748239 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:51.748207    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xb8c\" (UniqueName: \"kubernetes.io/projected/f88674d2-586a-4fd9-ac95-711f572c5dc1-kube-api-access-2xb8c\") pod \"openshift-lws-operator-fd8ccff4c-pmlf6\" (UID: \"f88674d2-586a-4fd9-ac95-711f572c5dc1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pmlf6"
Jul 28 22:12:51.875887 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:51.875810    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pmlf6"
Jul 28 22:12:52.004984 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:52.004956    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pmlf6"]
Jul 28 22:12:52.007836 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:12:52.007808    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf88674d2_586a_4fd9_ac95_711f572c5dc1.slice/crio-ff1d5e66c5ae458a1f856da7ab45dbb24303c3e662de9b32c9e3e426bda60ef5 WatchSource:0}: Error finding container ff1d5e66c5ae458a1f856da7ab45dbb24303c3e662de9b32c9e3e426bda60ef5: Status 404 returned error can't find the container with id ff1d5e66c5ae458a1f856da7ab45dbb24303c3e662de9b32c9e3e426bda60ef5
Jul 28 22:12:52.684313 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:52.684278    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pmlf6" event={"ID":"f88674d2-586a-4fd9-ac95-711f572c5dc1","Type":"ContainerStarted","Data":"ff1d5e66c5ae458a1f856da7ab45dbb24303c3e662de9b32c9e3e426bda60ef5"}
Jul 28 22:12:55.697590 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:55.697553    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pmlf6" event={"ID":"f88674d2-586a-4fd9-ac95-711f572c5dc1","Type":"ContainerStarted","Data":"915bc614d88d6ec2a86331e92fbeb0ee6d4b2d1bb68479664adc28f8ae00d647"}
Jul 28 22:12:55.713929 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:12:55.713886    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pmlf6" podStartSLOduration=2.003050299 podStartE2EDuration="4.713869638s" totalImagesPullingTime="2.710819339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:12:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:12:55.712812825 +0000 UTC m=+473.092950470" watchObservedRunningTime="2026-07-28 22:12:55.713869638 +0000 UTC m=+473.094007282"
Jul 28 22:13:13.258170 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:13.258135    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6974ffcb4c-bsd4p"]
Jul 28 22:13:13.263814 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:13.263798    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6974ffcb4c-bsd4p"
Jul 28 22:13:13.273659 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:13.273640    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 28 22:13:13.273762 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:13.273664    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-67bxr\""
Jul 28 22:13:13.273922 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:13.273908    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 28 22:13:13.273989 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:13.273921    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 28 22:13:13.274035 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:13.274013    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 28 22:13:13.292551 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:13.292525    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6974ffcb4c-bsd4p"]
Jul 28 22:13:13.408118 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:13.408081    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dfe553a9-7cf9-4715-865d-77d843be49a3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6974ffcb4c-bsd4p\" (UID: \"dfe553a9-7cf9-4715-865d-77d843be49a3\") " pod="opendatahub/opendatahub-operator-controller-manager-6974ffcb4c-bsd4p"
Jul 28 22:13:13.408276 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:13.408133    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfe553a9-7cf9-4715-865d-77d843be49a3-tmp\") pod \"opendatahub-operator-controller-manager-6974ffcb4c-bsd4p\" (UID: \"dfe553a9-7cf9-4715-865d-77d843be49a3\") " pod="opendatahub/opendatahub-operator-controller-manager-6974ffcb4c-bsd4p"
Jul 28 22:13:13.408276 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:13.408185    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pztsb\" (UniqueName: \"kubernetes.io/projected/dfe553a9-7cf9-4715-865d-77d843be49a3-kube-api-access-pztsb\") pod \"opendatahub-operator-controller-manager-6974ffcb4c-bsd4p\" (UID: \"dfe553a9-7cf9-4715-865d-77d843be49a3\") " pod="opendatahub/opendatahub-operator-controller-manager-6974ffcb4c-bsd4p"
Jul 28 22:13:13.408276 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:13.408232    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dfe553a9-7cf9-4715-865d-77d843be49a3-manifests\") pod \"opendatahub-operator-controller-manager-6974ffcb4c-bsd4p\" (UID: \"dfe553a9-7cf9-4715-865d-77d843be49a3\") " pod="opendatahub/opendatahub-operator-controller-manager-6974ffcb4c-bsd4p"
Jul 28 22:13:13.408276 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:13.408267    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dfe553a9-7cf9-4715-865d-77d843be49a3-webhook-cert\") pod \"opendatahub-operator-controller-manager-6974ffcb4c-bsd4p\" (UID: \"dfe553a9-7cf9-4715-865d-77d843be49a3\") " pod="opendatahub/opendatahub-operator-controller-manager-6974ffcb4c-bsd4p"
Jul 28 22:13:13.508740 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:13.508649    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dfe553a9-7cf9-4715-865d-77d843be49a3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6974ffcb4c-bsd4p\" (UID: \"dfe553a9-7cf9-4715-865d-77d843be49a3\") " pod="opendatahub/opendatahub-operator-controller-manager-6974ffcb4c-bsd4p"
Jul 28 22:13:13.508740 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:13.508709    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfe553a9-7cf9-4715-865d-77d843be49a3-tmp\") pod \"opendatahub-operator-controller-manager-6974ffcb4c-bsd4p\" (UID: \"dfe553a9-7cf9-4715-865d-77d843be49a3\") " pod="opendatahub/opendatahub-operator-controller-manager-6974ffcb4c-bsd4p"
Jul 28 22:13:13.508740 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:13.508737    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pztsb\" (UniqueName: \"kubernetes.io/projected/dfe553a9-7cf9-4715-865d-77d843be49a3-kube-api-access-pztsb\") pod \"opendatahub-operator-controller-manager-6974ffcb4c-bsd4p\" (UID: \"dfe553a9-7cf9-4715-865d-77d843be49a3\") " pod="opendatahub/opendatahub-operator-controller-manager-6974ffcb4c-bsd4p"
Jul 28 22:13:13.508990 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:13.508762    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dfe553a9-7cf9-4715-865d-77d843be49a3-manifests\") pod \"opendatahub-operator-controller-manager-6974ffcb4c-bsd4p\" (UID: \"dfe553a9-7cf9-4715-865d-77d843be49a3\") " pod="opendatahub/opendatahub-operator-controller-manager-6974ffcb4c-bsd4p"
Jul 28 22:13:13.508990 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:13.508799    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dfe553a9-7cf9-4715-865d-77d843be49a3-webhook-cert\") pod \"opendatahub-operator-controller-manager-6974ffcb4c-bsd4p\" (UID: \"dfe553a9-7cf9-4715-865d-77d843be49a3\") " pod="opendatahub/opendatahub-operator-controller-manager-6974ffcb4c-bsd4p"
Jul 28 22:13:13.509212 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:13.509191    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dfe553a9-7cf9-4715-865d-77d843be49a3-manifests\") pod \"opendatahub-operator-controller-manager-6974ffcb4c-bsd4p\" (UID: \"dfe553a9-7cf9-4715-865d-77d843be49a3\") " pod="opendatahub/opendatahub-operator-controller-manager-6974ffcb4c-bsd4p"
Jul 28 22:13:13.510995 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:13.510966    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfe553a9-7cf9-4715-865d-77d843be49a3-tmp\") pod \"opendatahub-operator-controller-manager-6974ffcb4c-bsd4p\" (UID: \"dfe553a9-7cf9-4715-865d-77d843be49a3\") " pod="opendatahub/opendatahub-operator-controller-manager-6974ffcb4c-bsd4p"
Jul 28 22:13:13.511239 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:13.511219    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dfe553a9-7cf9-4715-865d-77d843be49a3-webhook-cert\") pod \"opendatahub-operator-controller-manager-6974ffcb4c-bsd4p\" (UID: \"dfe553a9-7cf9-4715-865d-77d843be49a3\") " pod="opendatahub/opendatahub-operator-controller-manager-6974ffcb4c-bsd4p"
Jul 28 22:13:13.511334 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:13.511314    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dfe553a9-7cf9-4715-865d-77d843be49a3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6974ffcb4c-bsd4p\" (UID: \"dfe553a9-7cf9-4715-865d-77d843be49a3\") " pod="opendatahub/opendatahub-operator-controller-manager-6974ffcb4c-bsd4p"
Jul 28 22:13:13.518791 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:13.518768    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pztsb\" (UniqueName: \"kubernetes.io/projected/dfe553a9-7cf9-4715-865d-77d843be49a3-kube-api-access-pztsb\") pod \"opendatahub-operator-controller-manager-6974ffcb4c-bsd4p\" (UID: \"dfe553a9-7cf9-4715-865d-77d843be49a3\") " pod="opendatahub/opendatahub-operator-controller-manager-6974ffcb4c-bsd4p"
Jul 28 22:13:13.573407 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:13.573376    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6974ffcb4c-bsd4p"
Jul 28 22:13:13.708884 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:13.708850    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6974ffcb4c-bsd4p"]
Jul 28 22:13:13.712180 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:13:13.712155    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfe553a9_7cf9_4715_865d_77d843be49a3.slice/crio-23611ed0ef27a7ff105871924d259c7d361a890976fb0411fc16fdf257d33a30 WatchSource:0}: Error finding container 23611ed0ef27a7ff105871924d259c7d361a890976fb0411fc16fdf257d33a30: Status 404 returned error can't find the container with id 23611ed0ef27a7ff105871924d259c7d361a890976fb0411fc16fdf257d33a30
Jul 28 22:13:13.763798 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:13.763720    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6974ffcb4c-bsd4p" event={"ID":"dfe553a9-7cf9-4715-865d-77d843be49a3","Type":"ContainerStarted","Data":"23611ed0ef27a7ff105871924d259c7d361a890976fb0411fc16fdf257d33a30"}
Jul 28 22:13:16.778621 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:16.778586    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6974ffcb4c-bsd4p" event={"ID":"dfe553a9-7cf9-4715-865d-77d843be49a3","Type":"ContainerStarted","Data":"d01d5507a2665142faf77a11a078ac4cd79ed2cb6f46505aa8bc47f95f6611e7"}
Jul 28 22:13:18.788240 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:18.788205    2563 generic.go:356] "Generic (PLEG): container finished" podID="dfe553a9-7cf9-4715-865d-77d843be49a3" containerID="d01d5507a2665142faf77a11a078ac4cd79ed2cb6f46505aa8bc47f95f6611e7" exitCode=0
Jul 28 22:13:18.788637 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:18.788258    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6974ffcb4c-bsd4p" event={"ID":"dfe553a9-7cf9-4715-865d-77d843be49a3","Type":"ContainerDied","Data":"d01d5507a2665142faf77a11a078ac4cd79ed2cb6f46505aa8bc47f95f6611e7"}
Jul 28 22:13:19.793507 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:19.793476    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6974ffcb4c-bsd4p" event={"ID":"dfe553a9-7cf9-4715-865d-77d843be49a3","Type":"ContainerStarted","Data":"79360484c1e66c68eb15e5ee35b7915706d1cbc36731db789948fc36414eb28b"}
Jul 28 22:13:19.793942 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:19.793691    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6974ffcb4c-bsd4p"
Jul 28 22:13:19.817147 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:19.817099    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6974ffcb4c-bsd4p" podStartSLOduration=2.44750172 podStartE2EDuration="6.817085097s" totalImagesPullingTime="3.282818953s" totalInitContainerRuntime="1.086764424s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:13:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:13:19.8167217 +0000 UTC m=+497.196859343" watchObservedRunningTime="2026-07-28 22:13:19.817085097 +0000 UTC m=+497.197222741"
Jul 28 22:13:30.799584 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:30.799553    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6974ffcb4c-bsd4p"
Jul 28 22:13:33.366628 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:33.366590    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-9899fcb89-58jft"]
Jul 28 22:13:33.369853 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:33.369831    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9899fcb89-58jft"
Jul 28 22:13:33.373285 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:33.373266    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-qrtnr\""
Jul 28 22:13:33.373610 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:33.373585    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 22:13:33.374220 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:33.374205    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 28 22:13:33.374296 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:33.374278    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 28 22:13:33.374341 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:33.374287    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 22:13:33.403163 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:33.403137    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9899fcb89-58jft"]
Jul 28 22:13:33.465735 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:33.465701    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da9ae5dc-c3b4-459b-a25f-76b16e4ea340-tls-certs\") pod \"kube-auth-proxy-9899fcb89-58jft\" (UID: \"da9ae5dc-c3b4-459b-a25f-76b16e4ea340\") " pod="openshift-ingress/kube-auth-proxy-9899fcb89-58jft"
Jul 28 22:13:33.465882 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:33.465769    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da9ae5dc-c3b4-459b-a25f-76b16e4ea340-tmp\") pod \"kube-auth-proxy-9899fcb89-58jft\" (UID: \"da9ae5dc-c3b4-459b-a25f-76b16e4ea340\") " pod="openshift-ingress/kube-auth-proxy-9899fcb89-58jft"
Jul 28 22:13:33.465882 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:33.465830    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md5w8\" (UniqueName: \"kubernetes.io/projected/da9ae5dc-c3b4-459b-a25f-76b16e4ea340-kube-api-access-md5w8\") pod \"kube-auth-proxy-9899fcb89-58jft\" (UID: \"da9ae5dc-c3b4-459b-a25f-76b16e4ea340\") " pod="openshift-ingress/kube-auth-proxy-9899fcb89-58jft"
Jul 28 22:13:33.567165 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:33.567127    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da9ae5dc-c3b4-459b-a25f-76b16e4ea340-tls-certs\") pod \"kube-auth-proxy-9899fcb89-58jft\" (UID: \"da9ae5dc-c3b4-459b-a25f-76b16e4ea340\") " pod="openshift-ingress/kube-auth-proxy-9899fcb89-58jft"
Jul 28 22:13:33.567337 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:33.567237    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da9ae5dc-c3b4-459b-a25f-76b16e4ea340-tmp\") pod \"kube-auth-proxy-9899fcb89-58jft\" (UID: \"da9ae5dc-c3b4-459b-a25f-76b16e4ea340\") " pod="openshift-ingress/kube-auth-proxy-9899fcb89-58jft"
Jul 28 22:13:33.567337 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:33.567276    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md5w8\" (UniqueName: \"kubernetes.io/projected/da9ae5dc-c3b4-459b-a25f-76b16e4ea340-kube-api-access-md5w8\") pod \"kube-auth-proxy-9899fcb89-58jft\" (UID: \"da9ae5dc-c3b4-459b-a25f-76b16e4ea340\") " pod="openshift-ingress/kube-auth-proxy-9899fcb89-58jft"
Jul 28 22:13:33.567480 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:13:33.567278    2563 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 28 22:13:33.567480 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:13:33.567437    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da9ae5dc-c3b4-459b-a25f-76b16e4ea340-tls-certs podName:da9ae5dc-c3b4-459b-a25f-76b16e4ea340 nodeName:}" failed. No retries permitted until 2026-07-28 22:13:34.067413364 +0000 UTC m=+511.447550989 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/da9ae5dc-c3b4-459b-a25f-76b16e4ea340-tls-certs") pod "kube-auth-proxy-9899fcb89-58jft" (UID: "da9ae5dc-c3b4-459b-a25f-76b16e4ea340") : secret "kube-auth-proxy-tls" not found
Jul 28 22:13:33.569424 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:33.569402    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da9ae5dc-c3b4-459b-a25f-76b16e4ea340-tmp\") pod \"kube-auth-proxy-9899fcb89-58jft\" (UID: \"da9ae5dc-c3b4-459b-a25f-76b16e4ea340\") " pod="openshift-ingress/kube-auth-proxy-9899fcb89-58jft"
Jul 28 22:13:33.589166 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:33.589136    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-md5w8\" (UniqueName: \"kubernetes.io/projected/da9ae5dc-c3b4-459b-a25f-76b16e4ea340-kube-api-access-md5w8\") pod \"kube-auth-proxy-9899fcb89-58jft\" (UID: \"da9ae5dc-c3b4-459b-a25f-76b16e4ea340\") " pod="openshift-ingress/kube-auth-proxy-9899fcb89-58jft"
Jul 28 22:13:34.071384 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:34.071323    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da9ae5dc-c3b4-459b-a25f-76b16e4ea340-tls-certs\") pod \"kube-auth-proxy-9899fcb89-58jft\" (UID: \"da9ae5dc-c3b4-459b-a25f-76b16e4ea340\") " pod="openshift-ingress/kube-auth-proxy-9899fcb89-58jft"
Jul 28 22:13:34.073655 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:34.073626    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da9ae5dc-c3b4-459b-a25f-76b16e4ea340-tls-certs\") pod \"kube-auth-proxy-9899fcb89-58jft\" (UID: \"da9ae5dc-c3b4-459b-a25f-76b16e4ea340\") " pod="openshift-ingress/kube-auth-proxy-9899fcb89-58jft"
Jul 28 22:13:34.278950 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:34.278904    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9899fcb89-58jft"
Jul 28 22:13:34.419800 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:34.419679    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9899fcb89-58jft"]
Jul 28 22:13:34.422463 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:13:34.422438    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda9ae5dc_c3b4_459b_a25f_76b16e4ea340.slice/crio-af97415ad042cc32f50a685347d053298026c2239bdba0fbe992ba015b75703f WatchSource:0}: Error finding container af97415ad042cc32f50a685347d053298026c2239bdba0fbe992ba015b75703f: Status 404 returned error can't find the container with id af97415ad042cc32f50a685347d053298026c2239bdba0fbe992ba015b75703f
Jul 28 22:13:34.505756 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:34.505720    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mwl94"]
Jul 28 22:13:34.509981 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:34.509965    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-mwl94"
Jul 28 22:13:34.514900 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:34.514877    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 28 22:13:34.515009 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:34.514909    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-cmw92\""
Jul 28 22:13:34.527145 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:34.527126    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mwl94"]
Jul 28 22:13:34.575996 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:34.575966    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9072fe04-b34c-4640-81ba-da5e1b87134a-tls-certs\") pod \"model-serving-api-54d89bd5fc-mwl94\" (UID: \"9072fe04-b34c-4640-81ba-da5e1b87134a\") " pod="opendatahub/model-serving-api-54d89bd5fc-mwl94"
Jul 28 22:13:34.576128 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:34.576046    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk9pl\" (UniqueName: \"kubernetes.io/projected/9072fe04-b34c-4640-81ba-da5e1b87134a-kube-api-access-sk9pl\") pod \"model-serving-api-54d89bd5fc-mwl94\" (UID: \"9072fe04-b34c-4640-81ba-da5e1b87134a\") " pod="opendatahub/model-serving-api-54d89bd5fc-mwl94"
Jul 28 22:13:34.676480 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:34.676392    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk9pl\" (UniqueName: \"kubernetes.io/projected/9072fe04-b34c-4640-81ba-da5e1b87134a-kube-api-access-sk9pl\") pod \"model-serving-api-54d89bd5fc-mwl94\" (UID: \"9072fe04-b34c-4640-81ba-da5e1b87134a\") " pod="opendatahub/model-serving-api-54d89bd5fc-mwl94"
Jul 28 22:13:34.676480 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:34.676454    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9072fe04-b34c-4640-81ba-da5e1b87134a-tls-certs\") pod \"model-serving-api-54d89bd5fc-mwl94\" (UID: \"9072fe04-b34c-4640-81ba-da5e1b87134a\") " pod="opendatahub/model-serving-api-54d89bd5fc-mwl94"
Jul 28 22:13:34.678833 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:34.678807    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9072fe04-b34c-4640-81ba-da5e1b87134a-tls-certs\") pod \"model-serving-api-54d89bd5fc-mwl94\" (UID: \"9072fe04-b34c-4640-81ba-da5e1b87134a\") " pod="opendatahub/model-serving-api-54d89bd5fc-mwl94"
Jul 28 22:13:34.687526 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:34.687498    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk9pl\" (UniqueName: \"kubernetes.io/projected/9072fe04-b34c-4640-81ba-da5e1b87134a-kube-api-access-sk9pl\") pod \"model-serving-api-54d89bd5fc-mwl94\" (UID: \"9072fe04-b34c-4640-81ba-da5e1b87134a\") " pod="opendatahub/model-serving-api-54d89bd5fc-mwl94"
Jul 28 22:13:34.819984 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:34.819955    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-mwl94"
Jul 28 22:13:34.851173 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:34.851141    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9899fcb89-58jft" event={"ID":"da9ae5dc-c3b4-459b-a25f-76b16e4ea340","Type":"ContainerStarted","Data":"af97415ad042cc32f50a685347d053298026c2239bdba0fbe992ba015b75703f"}
Jul 28 22:13:34.947825 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:34.947791    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mwl94"]
Jul 28 22:13:34.950533 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:13:34.950509    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9072fe04_b34c_4640_81ba_da5e1b87134a.slice/crio-2b2adcf2df006668a25e13a56a9451a43f3a21570956fde24d50fe6b1d2a11c2 WatchSource:0}: Error finding container 2b2adcf2df006668a25e13a56a9451a43f3a21570956fde24d50fe6b1d2a11c2: Status 404 returned error can't find the container with id 2b2adcf2df006668a25e13a56a9451a43f3a21570956fde24d50fe6b1d2a11c2
Jul 28 22:13:35.858687 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:35.858639    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-mwl94" event={"ID":"9072fe04-b34c-4640-81ba-da5e1b87134a","Type":"ContainerStarted","Data":"2b2adcf2df006668a25e13a56a9451a43f3a21570956fde24d50fe6b1d2a11c2"}
Jul 28 22:13:37.870385 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:37.870338    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9899fcb89-58jft" event={"ID":"da9ae5dc-c3b4-459b-a25f-76b16e4ea340","Type":"ContainerStarted","Data":"00bec42d768b73d5c91cbe6f76b11be2db915793ee90980079bd133e91e9a532"}
Jul 28 22:13:37.871791 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:37.871768    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-mwl94" event={"ID":"9072fe04-b34c-4640-81ba-da5e1b87134a","Type":"ContainerStarted","Data":"67a3aa18cbdbb36a300ed3d64f4e1bd24d231375d813616fd8057eee0dd9a8d8"}
Jul 28 22:13:37.871932 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:37.871900    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-mwl94"
Jul 28 22:13:37.885891 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:37.885849    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-9899fcb89-58jft" podStartSLOduration=2.028136031 podStartE2EDuration="4.885833244s" totalImagesPullingTime="2.857697213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:13:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:13:37.8854336 +0000 UTC m=+515.265571245" watchObservedRunningTime="2026-07-28 22:13:37.885833244 +0000 UTC m=+515.265970891"
Jul 28 22:13:37.900443 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:37.900398    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-mwl94" podStartSLOduration=1.5775793340000002 podStartE2EDuration="3.900383425s" totalImagesPullingTime="2.322804091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:13:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:13:37.89993082 +0000 UTC m=+515.280068465" watchObservedRunningTime="2026-07-28 22:13:37.900383425 +0000 UTC m=+515.280521066"
Jul 28 22:13:40.289912 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:40.289874    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-fn9t5"]
Jul 28 22:13:40.293149 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:40.293133    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fn9t5"
Jul 28 22:13:40.296184 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:40.296160    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 28 22:13:40.296444 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:40.296422    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 28 22:13:40.296444 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:40.296437    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-kg6rk\""
Jul 28 22:13:40.296548 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:40.296469    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 28 22:13:40.296950 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:40.296934    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 28 22:13:40.305551 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:40.305526    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-fn9t5"]
Jul 28 22:13:40.326735 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:40.326710    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/12c068fd-02c5-4ff8-8ec5-fe35c37d3139-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-fn9t5\" (UID: \"12c068fd-02c5-4ff8-8ec5-fe35c37d3139\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fn9t5"
Jul 28 22:13:40.326875 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:40.326740    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/12c068fd-02c5-4ff8-8ec5-fe35c37d3139-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-fn9t5\" (UID: \"12c068fd-02c5-4ff8-8ec5-fe35c37d3139\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fn9t5"
Jul 28 22:13:40.326875 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:40.326779    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12c068fd-02c5-4ff8-8ec5-fe35c37d3139-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-fn9t5\" (UID: \"12c068fd-02c5-4ff8-8ec5-fe35c37d3139\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fn9t5"
Jul 28 22:13:40.326875 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:40.326837    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/12c068fd-02c5-4ff8-8ec5-fe35c37d3139-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-fn9t5\" (UID: \"12c068fd-02c5-4ff8-8ec5-fe35c37d3139\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fn9t5"
Jul 28 22:13:40.326979 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:40.326881    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/12c068fd-02c5-4ff8-8ec5-fe35c37d3139-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-fn9t5\" (UID: \"12c068fd-02c5-4ff8-8ec5-fe35c37d3139\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fn9t5"
Jul 28 22:13:40.326979 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:40.326933    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ldm5\" (UniqueName: \"kubernetes.io/projected/12c068fd-02c5-4ff8-8ec5-fe35c37d3139-kube-api-access-4ldm5\") pod \"istiod-openshift-gateway-5677ccb49b-fn9t5\" (UID: \"12c068fd-02c5-4ff8-8ec5-fe35c37d3139\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fn9t5"
Jul 28 22:13:40.327040 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:40.326969    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/12c068fd-02c5-4ff8-8ec5-fe35c37d3139-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-fn9t5\" (UID: \"12c068fd-02c5-4ff8-8ec5-fe35c37d3139\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fn9t5"
Jul 28 22:13:40.428105 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:40.428074    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12c068fd-02c5-4ff8-8ec5-fe35c37d3139-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-fn9t5\" (UID: \"12c068fd-02c5-4ff8-8ec5-fe35c37d3139\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fn9t5"
Jul 28 22:13:40.428105 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:40.428109    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/12c068fd-02c5-4ff8-8ec5-fe35c37d3139-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-fn9t5\" (UID: \"12c068fd-02c5-4ff8-8ec5-fe35c37d3139\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fn9t5"
Jul 28 22:13:40.428346 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:40.428135    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/12c068fd-02c5-4ff8-8ec5-fe35c37d3139-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-fn9t5\" (UID: \"12c068fd-02c5-4ff8-8ec5-fe35c37d3139\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fn9t5"
Jul 28 22:13:40.428346 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:40.428164    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ldm5\" (UniqueName: \"kubernetes.io/projected/12c068fd-02c5-4ff8-8ec5-fe35c37d3139-kube-api-access-4ldm5\") pod \"istiod-openshift-gateway-5677ccb49b-fn9t5\" (UID: \"12c068fd-02c5-4ff8-8ec5-fe35c37d3139\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fn9t5"
Jul 28 22:13:40.428346 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:40.428186    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/12c068fd-02c5-4ff8-8ec5-fe35c37d3139-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-fn9t5\" (UID: \"12c068fd-02c5-4ff8-8ec5-fe35c37d3139\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fn9t5"
Jul 28 22:13:40.428346 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:40.428219    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/12c068fd-02c5-4ff8-8ec5-fe35c37d3139-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-fn9t5\" (UID: \"12c068fd-02c5-4ff8-8ec5-fe35c37d3139\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fn9t5"
Jul 28 22:13:40.428346 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:40.428234    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/12c068fd-02c5-4ff8-8ec5-fe35c37d3139-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-fn9t5\" (UID: \"12c068fd-02c5-4ff8-8ec5-fe35c37d3139\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fn9t5"
Jul 28 22:13:40.428905 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:40.428879    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/12c068fd-02c5-4ff8-8ec5-fe35c37d3139-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-fn9t5\" (UID: \"12c068fd-02c5-4ff8-8ec5-fe35c37d3139\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fn9t5"
Jul 28 22:13:40.430538 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:40.430515    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/12c068fd-02c5-4ff8-8ec5-fe35c37d3139-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-fn9t5\" (UID: \"12c068fd-02c5-4ff8-8ec5-fe35c37d3139\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fn9t5"
Jul 28 22:13:40.430618 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:40.430526    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/12c068fd-02c5-4ff8-8ec5-fe35c37d3139-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-fn9t5\" (UID: \"12c068fd-02c5-4ff8-8ec5-fe35c37d3139\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fn9t5"
Jul 28 22:13:40.430722 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:40.430698    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/12c068fd-02c5-4ff8-8ec5-fe35c37d3139-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-fn9t5\" (UID: \"12c068fd-02c5-4ff8-8ec5-fe35c37d3139\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fn9t5"
Jul 28 22:13:40.430761 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:40.430705    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/12c068fd-02c5-4ff8-8ec5-fe35c37d3139-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-fn9t5\" (UID: \"12c068fd-02c5-4ff8-8ec5-fe35c37d3139\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fn9t5"
Jul 28 22:13:40.436602 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:40.436578    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12c068fd-02c5-4ff8-8ec5-fe35c37d3139-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-fn9t5\" (UID: \"12c068fd-02c5-4ff8-8ec5-fe35c37d3139\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fn9t5"
Jul 28 22:13:40.436906 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:40.436889    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ldm5\" (UniqueName: \"kubernetes.io/projected/12c068fd-02c5-4ff8-8ec5-fe35c37d3139-kube-api-access-4ldm5\") pod \"istiod-openshift-gateway-5677ccb49b-fn9t5\" (UID: \"12c068fd-02c5-4ff8-8ec5-fe35c37d3139\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fn9t5"
Jul 28 22:13:40.602193 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:40.602112    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fn9t5"
Jul 28 22:13:40.738915 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:40.738893    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-fn9t5"]
Jul 28 22:13:40.740797 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:13:40.740766    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12c068fd_02c5_4ff8_8ec5_fe35c37d3139.slice/crio-bf1f099d90a39dc2eb34817c5430c8f8c60e777323d7bca2243436a12849dabf WatchSource:0}: Error finding container bf1f099d90a39dc2eb34817c5430c8f8c60e777323d7bca2243436a12849dabf: Status 404 returned error can't find the container with id bf1f099d90a39dc2eb34817c5430c8f8c60e777323d7bca2243436a12849dabf
Jul 28 22:13:40.885426 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:40.885321    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fn9t5" event={"ID":"12c068fd-02c5-4ff8-8ec5-fe35c37d3139","Type":"ContainerStarted","Data":"bf1f099d90a39dc2eb34817c5430c8f8c60e777323d7bca2243436a12849dabf"}
Jul 28 22:13:43.273106 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:43.273057    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 22:13:43.273524 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:43.273148    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 22:13:43.902806 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:43.902750    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fn9t5" event={"ID":"12c068fd-02c5-4ff8-8ec5-fe35c37d3139","Type":"ContainerStarted","Data":"2be24c305991bba6700c522d61cd30b0da7be2a00124eb83351694f48df5ba62"}
Jul 28 22:13:43.903014 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:43.902914    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fn9t5"
Jul 28 22:13:43.949130 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:43.949073    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fn9t5" podStartSLOduration=1.419253634 podStartE2EDuration="3.949056775s" totalImagesPullingTime="2.529803141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:13:43.94568993 +0000 UTC m=+521.325827578" watchObservedRunningTime="2026-07-28 22:13:43.949056775 +0000 UTC m=+521.329194419"
Jul 28 22:13:44.908528 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:44.908501    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fn9t5"
Jul 28 22:13:48.879432 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:13:48.879402    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-mwl94"
Jul 28 22:14:34.125105 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:34.125025    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wnpjz"]
Jul 28 22:14:34.127314 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:34.127298    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wnpjz"
Jul 28 22:14:34.130924 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:34.130902    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-6mmkx\""
Jul 28 22:14:34.131060 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:34.130902    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 28 22:14:34.131951 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:34.131924    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 28 22:14:34.145485 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:34.145455    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wnpjz"]
Jul 28 22:14:34.163613 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:34.163585    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95spj\" (UniqueName: \"kubernetes.io/projected/c5b38505-a383-45af-86c1-bf77dc097ca5-kube-api-access-95spj\") pod \"kuadrant-operator-catalog-wnpjz\" (UID: \"c5b38505-a383-45af-86c1-bf77dc097ca5\") " pod="kuadrant-system/kuadrant-operator-catalog-wnpjz"
Jul 28 22:14:34.264497 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:34.264464    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95spj\" (UniqueName: \"kubernetes.io/projected/c5b38505-a383-45af-86c1-bf77dc097ca5-kube-api-access-95spj\") pod \"kuadrant-operator-catalog-wnpjz\" (UID: \"c5b38505-a383-45af-86c1-bf77dc097ca5\") " pod="kuadrant-system/kuadrant-operator-catalog-wnpjz"
Jul 28 22:14:34.273019 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:34.272989    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95spj\" (UniqueName: \"kubernetes.io/projected/c5b38505-a383-45af-86c1-bf77dc097ca5-kube-api-access-95spj\") pod \"kuadrant-operator-catalog-wnpjz\" (UID: \"c5b38505-a383-45af-86c1-bf77dc097ca5\") " pod="kuadrant-system/kuadrant-operator-catalog-wnpjz"
Jul 28 22:14:34.437165 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:34.437140    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wnpjz"
Jul 28 22:14:34.489657 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:34.489630    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wnpjz"]
Jul 28 22:14:34.563053 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:34.563025    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wnpjz"]
Jul 28 22:14:34.565618 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:14:34.565576    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5b38505_a383_45af_86c1_bf77dc097ca5.slice/crio-d3232d6d6d37e829ea6c3a0657e729358a572ad7a34520b53cd8d5dee6694221 WatchSource:0}: Error finding container d3232d6d6d37e829ea6c3a0657e729358a572ad7a34520b53cd8d5dee6694221: Status 404 returned error can't find the container with id d3232d6d6d37e829ea6c3a0657e729358a572ad7a34520b53cd8d5dee6694221
Jul 28 22:14:34.697437 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:34.697370    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jlv6n"]
Jul 28 22:14:34.700009 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:34.699991    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jlv6n"
Jul 28 22:14:34.708270 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:34.708249    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jlv6n"]
Jul 28 22:14:34.768423 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:34.768392    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2xgk\" (UniqueName: \"kubernetes.io/projected/be85f180-b55d-42d8-931c-fdef76639076-kube-api-access-q2xgk\") pod \"kuadrant-operator-catalog-jlv6n\" (UID: \"be85f180-b55d-42d8-931c-fdef76639076\") " pod="kuadrant-system/kuadrant-operator-catalog-jlv6n"
Jul 28 22:14:34.869201 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:34.869168    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2xgk\" (UniqueName: \"kubernetes.io/projected/be85f180-b55d-42d8-931c-fdef76639076-kube-api-access-q2xgk\") pod \"kuadrant-operator-catalog-jlv6n\" (UID: \"be85f180-b55d-42d8-931c-fdef76639076\") " pod="kuadrant-system/kuadrant-operator-catalog-jlv6n"
Jul 28 22:14:34.877635 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:34.877611    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2xgk\" (UniqueName: \"kubernetes.io/projected/be85f180-b55d-42d8-931c-fdef76639076-kube-api-access-q2xgk\") pod \"kuadrant-operator-catalog-jlv6n\" (UID: \"be85f180-b55d-42d8-931c-fdef76639076\") " pod="kuadrant-system/kuadrant-operator-catalog-jlv6n"
Jul 28 22:14:35.009992 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:35.009911    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jlv6n"
Jul 28 22:14:35.108985 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:35.108953    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wnpjz" event={"ID":"c5b38505-a383-45af-86c1-bf77dc097ca5","Type":"ContainerStarted","Data":"d3232d6d6d37e829ea6c3a0657e729358a572ad7a34520b53cd8d5dee6694221"}
Jul 28 22:14:35.143772 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:35.143751    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jlv6n"]
Jul 28 22:14:35.145765 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:14:35.145733    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe85f180_b55d_42d8_931c_fdef76639076.slice/crio-01e0ea89622e6b9fe6de599b023a87c6ff00c6945a0a91ba378224184bef79f7 WatchSource:0}: Error finding container 01e0ea89622e6b9fe6de599b023a87c6ff00c6945a0a91ba378224184bef79f7: Status 404 returned error can't find the container with id 01e0ea89622e6b9fe6de599b023a87c6ff00c6945a0a91ba378224184bef79f7
Jul 28 22:14:36.115772 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:36.115732    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-jlv6n" event={"ID":"be85f180-b55d-42d8-931c-fdef76639076","Type":"ContainerStarted","Data":"01e0ea89622e6b9fe6de599b023a87c6ff00c6945a0a91ba378224184bef79f7"}
Jul 28 22:14:37.121347 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:37.121283    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wnpjz" event={"ID":"c5b38505-a383-45af-86c1-bf77dc097ca5","Type":"ContainerStarted","Data":"3cdfb5289e580ed8a71382336ed118d59e1b28e97b4d1afd15779954def16fa9"}
Jul 28 22:14:37.121347 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:37.121323    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-wnpjz" podUID="c5b38505-a383-45af-86c1-bf77dc097ca5" containerName="registry-server" containerID="cri-o://3cdfb5289e580ed8a71382336ed118d59e1b28e97b4d1afd15779954def16fa9" gracePeriod=2
Jul 28 22:14:37.122703 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:37.122678    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-jlv6n" event={"ID":"be85f180-b55d-42d8-931c-fdef76639076","Type":"ContainerStarted","Data":"df43c1fdc2d9c0c1a541e422810404abce6bb7d35a923e7414bf02214a060b95"}
Jul 28 22:14:37.138066 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:37.138027    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-wnpjz" podStartSLOduration=1.29239506 podStartE2EDuration="3.138012074s" totalImagesPullingTime="1.845617014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:14:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:14:37.136452848 +0000 UTC m=+574.516590492" watchObservedRunningTime="2026-07-28 22:14:37.138012074 +0000 UTC m=+574.518149718"
Jul 28 22:14:37.151582 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:37.151544    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-jlv6n" podStartSLOduration=1.885401369 podStartE2EDuration="3.151530297s" totalImagesPullingTime="1.266128928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:14:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:14:37.150964602 +0000 UTC m=+574.531102247" watchObservedRunningTime="2026-07-28 22:14:37.151530297 +0000 UTC m=+574.531667941"
Jul 28 22:14:37.365515 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:37.365485    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wnpjz"
Jul 28 22:14:37.394761 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:37.394706    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95spj\" (UniqueName: \"kubernetes.io/projected/c5b38505-a383-45af-86c1-bf77dc097ca5-kube-api-access-95spj\") pod \"c5b38505-a383-45af-86c1-bf77dc097ca5\" (UID: \"c5b38505-a383-45af-86c1-bf77dc097ca5\") "
Jul 28 22:14:37.396888 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:37.396859    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5b38505-a383-45af-86c1-bf77dc097ca5-kube-api-access-95spj" (OuterVolumeSpecName: "kube-api-access-95spj") pod "c5b38505-a383-45af-86c1-bf77dc097ca5" (UID: "c5b38505-a383-45af-86c1-bf77dc097ca5"). InnerVolumeSpecName "kube-api-access-95spj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:14:37.495167 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:37.495140    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95spj\" (UniqueName: \"kubernetes.io/projected/c5b38505-a383-45af-86c1-bf77dc097ca5-kube-api-access-95spj\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:14:38.127224 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:38.127191    2563 generic.go:356] "Generic (PLEG): container finished" podID="c5b38505-a383-45af-86c1-bf77dc097ca5" containerID="3cdfb5289e580ed8a71382336ed118d59e1b28e97b4d1afd15779954def16fa9" exitCode=0
Jul 28 22:14:38.127637 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:38.127236    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wnpjz"
Jul 28 22:14:38.127637 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:38.127274    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wnpjz" event={"ID":"c5b38505-a383-45af-86c1-bf77dc097ca5","Type":"ContainerDied","Data":"3cdfb5289e580ed8a71382336ed118d59e1b28e97b4d1afd15779954def16fa9"}
Jul 28 22:14:38.127637 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:38.127312    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wnpjz" event={"ID":"c5b38505-a383-45af-86c1-bf77dc097ca5","Type":"ContainerDied","Data":"d3232d6d6d37e829ea6c3a0657e729358a572ad7a34520b53cd8d5dee6694221"}
Jul 28 22:14:38.127637 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:38.127332    2563 scope.go:117] "RemoveContainer" containerID="3cdfb5289e580ed8a71382336ed118d59e1b28e97b4d1afd15779954def16fa9"
Jul 28 22:14:38.136385 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:38.136363    2563 scope.go:117] "RemoveContainer" containerID="3cdfb5289e580ed8a71382336ed118d59e1b28e97b4d1afd15779954def16fa9"
Jul 28 22:14:38.136647 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:14:38.136631    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cdfb5289e580ed8a71382336ed118d59e1b28e97b4d1afd15779954def16fa9\": container with ID starting with 3cdfb5289e580ed8a71382336ed118d59e1b28e97b4d1afd15779954def16fa9 not found: ID does not exist" containerID="3cdfb5289e580ed8a71382336ed118d59e1b28e97b4d1afd15779954def16fa9"
Jul 28 22:14:38.136700 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:38.136655    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cdfb5289e580ed8a71382336ed118d59e1b28e97b4d1afd15779954def16fa9"} err="failed to get container status \"3cdfb5289e580ed8a71382336ed118d59e1b28e97b4d1afd15779954def16fa9\": rpc error: code = NotFound desc = could not find container \"3cdfb5289e580ed8a71382336ed118d59e1b28e97b4d1afd15779954def16fa9\": container with ID starting with 3cdfb5289e580ed8a71382336ed118d59e1b28e97b4d1afd15779954def16fa9 not found: ID does not exist"
Jul 28 22:14:38.147866 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:38.147845    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wnpjz"]
Jul 28 22:14:38.151715 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:38.151696    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wnpjz"]
Jul 28 22:14:39.131251 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:39.131195    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5b38505-a383-45af-86c1-bf77dc097ca5" path="/var/lib/kubelet/pods/c5b38505-a383-45af-86c1-bf77dc097ca5/volumes"
Jul 28 22:14:45.010596 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:45.010556    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-jlv6n"
Jul 28 22:14:45.010997 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:45.010629    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-jlv6n"
Jul 28 22:14:45.031796 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:45.031772    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-jlv6n"
Jul 28 22:14:45.183698 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:14:45.183672    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-jlv6n"
Jul 28 22:15:03.012869 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:03.012835    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mcj8p_7bab32b0-13c6-4055-916e-a7454bafb5b6/ovn-acl-logging/0.log"
Jul 28 22:15:03.013302 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:03.013157    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mcj8p_7bab32b0-13c6-4055-916e-a7454bafb5b6/ovn-acl-logging/0.log"
Jul 28 22:15:06.745664 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:06.745625    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7rkst"]
Jul 28 22:15:06.751191 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:06.745956    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5b38505-a383-45af-86c1-bf77dc097ca5" containerName="registry-server"
Jul 28 22:15:06.751191 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:06.745969    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5b38505-a383-45af-86c1-bf77dc097ca5" containerName="registry-server"
Jul 28 22:15:06.751191 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:06.746027    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5b38505-a383-45af-86c1-bf77dc097ca5" containerName="registry-server"
Jul 28 22:15:06.751681 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:06.751665    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7rkst"
Jul 28 22:15:06.754418 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:06.754400    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-km9fm\""
Jul 28 22:15:06.759732 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:06.759709    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7rkst"]
Jul 28 22:15:06.834682 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:06.834652    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh6jh\" (UniqueName: \"kubernetes.io/projected/01e11fb0-1943-4fd6-8beb-1146b9723726-kube-api-access-fh6jh\") pod \"authorino-operator-75cc8d5cb7-7rkst\" (UID: \"01e11fb0-1943-4fd6-8beb-1146b9723726\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7rkst"
Jul 28 22:15:06.935474 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:06.935441    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh6jh\" (UniqueName: \"kubernetes.io/projected/01e11fb0-1943-4fd6-8beb-1146b9723726-kube-api-access-fh6jh\") pod \"authorino-operator-75cc8d5cb7-7rkst\" (UID: \"01e11fb0-1943-4fd6-8beb-1146b9723726\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7rkst"
Jul 28 22:15:06.947586 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:06.947559    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh6jh\" (UniqueName: \"kubernetes.io/projected/01e11fb0-1943-4fd6-8beb-1146b9723726-kube-api-access-fh6jh\") pod \"authorino-operator-75cc8d5cb7-7rkst\" (UID: \"01e11fb0-1943-4fd6-8beb-1146b9723726\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7rkst"
Jul 28 22:15:07.062282 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:07.062203    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7rkst"
Jul 28 22:15:07.197722 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:07.197688    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7rkst"]
Jul 28 22:15:07.201534 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:15:07.201504    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01e11fb0_1943_4fd6_8beb_1146b9723726.slice/crio-a385bde02dc093be6ffab7d498bf7977d8182a71865b645785ea0dbf9aaebfd0 WatchSource:0}: Error finding container a385bde02dc093be6ffab7d498bf7977d8182a71865b645785ea0dbf9aaebfd0: Status 404 returned error can't find the container with id a385bde02dc093be6ffab7d498bf7977d8182a71865b645785ea0dbf9aaebfd0
Jul 28 22:15:07.259383 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:07.259338    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7rkst" event={"ID":"01e11fb0-1943-4fd6-8beb-1146b9723726","Type":"ContainerStarted","Data":"a385bde02dc093be6ffab7d498bf7977d8182a71865b645785ea0dbf9aaebfd0"}
Jul 28 22:15:10.274220 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:10.274185    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wd6rb"]
Jul 28 22:15:10.277603 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:10.277580    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wd6rb"
Jul 28 22:15:10.277749 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:10.277626    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7rkst" event={"ID":"01e11fb0-1943-4fd6-8beb-1146b9723726","Type":"ContainerStarted","Data":"7f22e50a2a4de1b915ad0dc6cd36f698351c1759028ae82a3f0a43fbe06ba8b2"}
Jul 28 22:15:10.280589 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:10.280568    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-pknb7\""
Jul 28 22:15:10.292065 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:10.292044    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wd6rb"]
Jul 28 22:15:10.335216 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:10.335165    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7rkst" podStartSLOduration=2.068033997 podStartE2EDuration="4.335149247s" totalImagesPullingTime="2.26711525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:15:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:15:10.333440189 +0000 UTC m=+607.713577832" watchObservedRunningTime="2026-07-28 22:15:10.335149247 +0000 UTC m=+607.715286891"
Jul 28 22:15:10.366621 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:10.366586    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e506de1-4737-4e71-9080-0c341876c96d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wd6rb\" (UID: \"4e506de1-4737-4e71-9080-0c341876c96d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wd6rb"
Jul 28 22:15:10.366771 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:10.366628    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptvmg\" (UniqueName: \"kubernetes.io/projected/4e506de1-4737-4e71-9080-0c341876c96d-kube-api-access-ptvmg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wd6rb\" (UID: \"4e506de1-4737-4e71-9080-0c341876c96d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wd6rb"
Jul 28 22:15:10.467313 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:10.467269    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e506de1-4737-4e71-9080-0c341876c96d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wd6rb\" (UID: \"4e506de1-4737-4e71-9080-0c341876c96d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wd6rb"
Jul 28 22:15:10.467313 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:10.467313    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptvmg\" (UniqueName: \"kubernetes.io/projected/4e506de1-4737-4e71-9080-0c341876c96d-kube-api-access-ptvmg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wd6rb\" (UID: \"4e506de1-4737-4e71-9080-0c341876c96d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wd6rb"
Jul 28 22:15:10.467704 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:10.467683    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e506de1-4737-4e71-9080-0c341876c96d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wd6rb\" (UID: \"4e506de1-4737-4e71-9080-0c341876c96d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wd6rb"
Jul 28 22:15:10.478001 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:10.477969    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptvmg\" (UniqueName: \"kubernetes.io/projected/4e506de1-4737-4e71-9080-0c341876c96d-kube-api-access-ptvmg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wd6rb\" (UID: \"4e506de1-4737-4e71-9080-0c341876c96d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wd6rb"
Jul 28 22:15:10.586882 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:10.586797    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wd6rb"
Jul 28 22:15:10.733161 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:10.733120    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wd6rb"]
Jul 28 22:15:10.736118 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:15:10.736091    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e506de1_4737_4e71_9080_0c341876c96d.slice/crio-dc3b9a98e0745842b837a1792af79666816af4e025b511c9cd90536ed74e3cd2 WatchSource:0}: Error finding container dc3b9a98e0745842b837a1792af79666816af4e025b511c9cd90536ed74e3cd2: Status 404 returned error can't find the container with id dc3b9a98e0745842b837a1792af79666816af4e025b511c9cd90536ed74e3cd2
Jul 28 22:15:11.283311 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:11.283279    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wd6rb" event={"ID":"4e506de1-4737-4e71-9080-0c341876c96d","Type":"ContainerStarted","Data":"dc3b9a98e0745842b837a1792af79666816af4e025b511c9cd90536ed74e3cd2"}
Jul 28 22:15:11.283773 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:11.283469    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7rkst"
Jul 28 22:15:15.303392 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:15.303329    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wd6rb" event={"ID":"4e506de1-4737-4e71-9080-0c341876c96d","Type":"ContainerStarted","Data":"d3b2fb084816435569fbef006ab220cff7ea7e673502bd3c0e9233f6e1fc4337"}
Jul 28 22:15:15.303778 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:15.303503    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wd6rb"
Jul 28 22:15:15.332691 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:15.332636    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wd6rb" podStartSLOduration=1.526708105 podStartE2EDuration="5.332617277s" totalImagesPullingTime="3.805909172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:15:15.331847091 +0000 UTC m=+612.711984734" watchObservedRunningTime="2026-07-28 22:15:15.332617277 +0000 UTC m=+612.712754922"
Jul 28 22:15:22.290805 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:22.290775    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7rkst"
Jul 28 22:15:26.309681 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:26.309609    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wd6rb"
Jul 28 22:15:28.129678 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:28.129641    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wd6rb"]
Jul 28 22:15:28.130113 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:28.129800    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wd6rb" podUID="4e506de1-4737-4e71-9080-0c341876c96d" containerName="manager" containerID="cri-o://d3b2fb084816435569fbef006ab220cff7ea7e673502bd3c0e9233f6e1fc4337" gracePeriod=2
Jul 28 22:15:28.140689 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:28.140662    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wd6rb"]
Jul 28 22:15:28.150705 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:15:28.150672    2563 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"4e506de1-4737-4e71-9080-0c341876c96d\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-wd6rb\": pods \"kuadrant-operator-controller-manager-8684c9ff49-wd6rb\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wd6rb"
Jul 28 22:15:28.159437 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:28.159411    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gwv9l"]
Jul 28 22:15:28.159962 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:28.159937    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e506de1-4737-4e71-9080-0c341876c96d" containerName="manager"
Jul 28 22:15:28.160048 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:28.159965    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e506de1-4737-4e71-9080-0c341876c96d" containerName="manager"
Jul 28 22:15:28.160084 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:28.160059    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e506de1-4737-4e71-9080-0c341876c96d" containerName="manager"
Jul 28 22:15:28.163250 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:28.163231    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gwv9l"
Jul 28 22:15:28.167527 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:15:28.167502    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wd6rb\" is forbidden: User \"system:node:ip-10-0-128-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-37.ec2.internal' and this object" podUID="4e506de1-4737-4e71-9080-0c341876c96d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wd6rb"
Jul 28 22:15:28.182428 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:28.182397    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gwv9l"]
Jul 28 22:15:28.217682 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:28.217653    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aea07d6f-bf31-4ede-a50e-48a209883395-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-gwv9l\" (UID: \"aea07d6f-bf31-4ede-a50e-48a209883395\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gwv9l"
Jul 28 22:15:28.217793 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:28.217720    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzx84\" (UniqueName: \"kubernetes.io/projected/aea07d6f-bf31-4ede-a50e-48a209883395-kube-api-access-wzx84\") pod \"kuadrant-operator-controller-manager-8684c9ff49-gwv9l\" (UID: \"aea07d6f-bf31-4ede-a50e-48a209883395\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gwv9l"
Jul 28 22:15:28.319023 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:28.318993    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzx84\" (UniqueName: \"kubernetes.io/projected/aea07d6f-bf31-4ede-a50e-48a209883395-kube-api-access-wzx84\") pod \"kuadrant-operator-controller-manager-8684c9ff49-gwv9l\" (UID: \"aea07d6f-bf31-4ede-a50e-48a209883395\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gwv9l"
Jul 28 22:15:28.319187 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:28.319056    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aea07d6f-bf31-4ede-a50e-48a209883395-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-gwv9l\" (UID: \"aea07d6f-bf31-4ede-a50e-48a209883395\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gwv9l"
Jul 28 22:15:28.319459 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:28.319439    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aea07d6f-bf31-4ede-a50e-48a209883395-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-gwv9l\" (UID: \"aea07d6f-bf31-4ede-a50e-48a209883395\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gwv9l"
Jul 28 22:15:28.327094 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:28.327071    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzx84\" (UniqueName: \"kubernetes.io/projected/aea07d6f-bf31-4ede-a50e-48a209883395-kube-api-access-wzx84\") pod \"kuadrant-operator-controller-manager-8684c9ff49-gwv9l\" (UID: \"aea07d6f-bf31-4ede-a50e-48a209883395\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gwv9l"
Jul 28 22:15:28.363936 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:28.363905    2563 generic.go:356] "Generic (PLEG): container finished" podID="4e506de1-4737-4e71-9080-0c341876c96d" containerID="d3b2fb084816435569fbef006ab220cff7ea7e673502bd3c0e9233f6e1fc4337" exitCode=0
Jul 28 22:15:28.364041 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:28.363984    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc3b9a98e0745842b837a1792af79666816af4e025b511c9cd90536ed74e3cd2"
Jul 28 22:15:28.374205 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:28.374188    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wd6rb"
Jul 28 22:15:28.376475 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:15:28.376447    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wd6rb\" is forbidden: User \"system:node:ip-10-0-128-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-37.ec2.internal' and this object" podUID="4e506de1-4737-4e71-9080-0c341876c96d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wd6rb"
Jul 28 22:15:28.420196 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:28.420140    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptvmg\" (UniqueName: \"kubernetes.io/projected/4e506de1-4737-4e71-9080-0c341876c96d-kube-api-access-ptvmg\") pod \"4e506de1-4737-4e71-9080-0c341876c96d\" (UID: \"4e506de1-4737-4e71-9080-0c341876c96d\") "
Jul 28 22:15:28.420289 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:28.420214    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e506de1-4737-4e71-9080-0c341876c96d-extensions-socket-volume\") pod \"4e506de1-4737-4e71-9080-0c341876c96d\" (UID: \"4e506de1-4737-4e71-9080-0c341876c96d\") "
Jul 28 22:15:28.420689 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:28.420665    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e506de1-4737-4e71-9080-0c341876c96d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4e506de1-4737-4e71-9080-0c341876c96d" (UID: "4e506de1-4737-4e71-9080-0c341876c96d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:15:28.422071 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:28.422049    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e506de1-4737-4e71-9080-0c341876c96d-kube-api-access-ptvmg" (OuterVolumeSpecName: "kube-api-access-ptvmg") pod "4e506de1-4737-4e71-9080-0c341876c96d" (UID: "4e506de1-4737-4e71-9080-0c341876c96d"). InnerVolumeSpecName "kube-api-access-ptvmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:15:28.515887 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:28.515860    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gwv9l"
Jul 28 22:15:28.520970 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:28.520951    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e506de1-4737-4e71-9080-0c341876c96d-extensions-socket-volume\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:15:28.521053 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:28.520973    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptvmg\" (UniqueName: \"kubernetes.io/projected/4e506de1-4737-4e71-9080-0c341876c96d-kube-api-access-ptvmg\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:15:28.650910 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:28.650874    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gwv9l"]
Jul 28 22:15:28.654230 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:15:28.654192    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea07d6f_bf31_4ede_a50e_48a209883395.slice/crio-37c66de2ec51e9c681c41b5f5eef8c096539d001f5b048892716e31d27a6a68a WatchSource:0}: Error finding container 37c66de2ec51e9c681c41b5f5eef8c096539d001f5b048892716e31d27a6a68a: Status 404 returned error can't find the container with id 37c66de2ec51e9c681c41b5f5eef8c096539d001f5b048892716e31d27a6a68a
Jul 28 22:15:29.129834 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:29.129803    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e506de1-4737-4e71-9080-0c341876c96d" path="/var/lib/kubelet/pods/4e506de1-4737-4e71-9080-0c341876c96d/volumes"
Jul 28 22:15:29.369732 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:29.369698    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wd6rb"
Jul 28 22:15:29.369732 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:29.369721    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gwv9l" event={"ID":"aea07d6f-bf31-4ede-a50e-48a209883395","Type":"ContainerStarted","Data":"715e5da2441edaaca227e6904cc70940ca32998380f19f81962ed2f6bde900ad"}
Jul 28 22:15:29.369926 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:29.369774    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gwv9l" event={"ID":"aea07d6f-bf31-4ede-a50e-48a209883395","Type":"ContainerStarted","Data":"37c66de2ec51e9c681c41b5f5eef8c096539d001f5b048892716e31d27a6a68a"}
Jul 28 22:15:29.369926 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:29.369801    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gwv9l"
Jul 28 22:15:29.398547 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:29.398455    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gwv9l" podStartSLOduration=1.398435438 podStartE2EDuration="1.398435438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:15:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:15:29.396595151 +0000 UTC m=+626.776732795" watchObservedRunningTime="2026-07-28 22:15:29.398435438 +0000 UTC m=+626.778573083"
Jul 28 22:15:40.375930 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:40.375892    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gwv9l"
Jul 28 22:15:43.852696 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:43.852655    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gwv9l"]
Jul 28 22:15:43.853105 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:43.852839    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gwv9l" podUID="aea07d6f-bf31-4ede-a50e-48a209883395" containerName="manager" containerID="cri-o://715e5da2441edaaca227e6904cc70940ca32998380f19f81962ed2f6bde900ad" gracePeriod=10
Jul 28 22:15:44.099785 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:44.099755    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gwv9l"
Jul 28 22:15:44.100469 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:44.100446    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q6s9l"]
Jul 28 22:15:44.100798 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:44.100780    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aea07d6f-bf31-4ede-a50e-48a209883395" containerName="manager"
Jul 28 22:15:44.100879 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:44.100800    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="aea07d6f-bf31-4ede-a50e-48a209883395" containerName="manager"
Jul 28 22:15:44.100987 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:44.100894    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="aea07d6f-bf31-4ede-a50e-48a209883395" containerName="manager"
Jul 28 22:15:44.102882 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:44.102836    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q6s9l"
Jul 28 22:15:44.118723 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:44.118702    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q6s9l"]
Jul 28 22:15:44.140601 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:44.140570    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aea07d6f-bf31-4ede-a50e-48a209883395-extensions-socket-volume\") pod \"aea07d6f-bf31-4ede-a50e-48a209883395\" (UID: \"aea07d6f-bf31-4ede-a50e-48a209883395\") "
Jul 28 22:15:44.140715 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:44.140615    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzx84\" (UniqueName: \"kubernetes.io/projected/aea07d6f-bf31-4ede-a50e-48a209883395-kube-api-access-wzx84\") pod \"aea07d6f-bf31-4ede-a50e-48a209883395\" (UID: \"aea07d6f-bf31-4ede-a50e-48a209883395\") "
Jul 28 22:15:44.140852 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:44.140820    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad2a4a21-a0b5-4b90-ae3d-c29e89c26d01-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-q6s9l\" (UID: \"ad2a4a21-a0b5-4b90-ae3d-c29e89c26d01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q6s9l"
Jul 28 22:15:44.140909 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:44.140862    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hg8d\" (UniqueName: \"kubernetes.io/projected/ad2a4a21-a0b5-4b90-ae3d-c29e89c26d01-kube-api-access-9hg8d\") pod \"kuadrant-operator-controller-manager-86878c769f-q6s9l\" (UID: \"ad2a4a21-a0b5-4b90-ae3d-c29e89c26d01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q6s9l"
Jul 28 22:15:44.141002 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:44.140986    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aea07d6f-bf31-4ede-a50e-48a209883395-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "aea07d6f-bf31-4ede-a50e-48a209883395" (UID: "aea07d6f-bf31-4ede-a50e-48a209883395"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:15:44.143214 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:44.143187    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aea07d6f-bf31-4ede-a50e-48a209883395-kube-api-access-wzx84" (OuterVolumeSpecName: "kube-api-access-wzx84") pod "aea07d6f-bf31-4ede-a50e-48a209883395" (UID: "aea07d6f-bf31-4ede-a50e-48a209883395"). InnerVolumeSpecName "kube-api-access-wzx84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:15:44.241287 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:44.241254    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad2a4a21-a0b5-4b90-ae3d-c29e89c26d01-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-q6s9l\" (UID: \"ad2a4a21-a0b5-4b90-ae3d-c29e89c26d01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q6s9l"
Jul 28 22:15:44.241463 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:44.241303    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hg8d\" (UniqueName: \"kubernetes.io/projected/ad2a4a21-a0b5-4b90-ae3d-c29e89c26d01-kube-api-access-9hg8d\") pod \"kuadrant-operator-controller-manager-86878c769f-q6s9l\" (UID: \"ad2a4a21-a0b5-4b90-ae3d-c29e89c26d01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q6s9l"
Jul 28 22:15:44.241463 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:44.241454    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aea07d6f-bf31-4ede-a50e-48a209883395-extensions-socket-volume\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:15:44.241571 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:44.241470    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzx84\" (UniqueName: \"kubernetes.io/projected/aea07d6f-bf31-4ede-a50e-48a209883395-kube-api-access-wzx84\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:15:44.241655 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:44.241634    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad2a4a21-a0b5-4b90-ae3d-c29e89c26d01-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-q6s9l\" (UID: \"ad2a4a21-a0b5-4b90-ae3d-c29e89c26d01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q6s9l"
Jul 28 22:15:44.251927 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:44.251901    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hg8d\" (UniqueName: \"kubernetes.io/projected/ad2a4a21-a0b5-4b90-ae3d-c29e89c26d01-kube-api-access-9hg8d\") pod \"kuadrant-operator-controller-manager-86878c769f-q6s9l\" (UID: \"ad2a4a21-a0b5-4b90-ae3d-c29e89c26d01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q6s9l"
Jul 28 22:15:44.412213 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:44.412124    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q6s9l"
Jul 28 22:15:44.438887 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:44.438862    2563 generic.go:356] "Generic (PLEG): container finished" podID="aea07d6f-bf31-4ede-a50e-48a209883395" containerID="715e5da2441edaaca227e6904cc70940ca32998380f19f81962ed2f6bde900ad" exitCode=0
Jul 28 22:15:44.439001 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:44.438927    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gwv9l" event={"ID":"aea07d6f-bf31-4ede-a50e-48a209883395","Type":"ContainerDied","Data":"715e5da2441edaaca227e6904cc70940ca32998380f19f81962ed2f6bde900ad"}
Jul 28 22:15:44.439066 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:44.438999    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gwv9l" event={"ID":"aea07d6f-bf31-4ede-a50e-48a209883395","Type":"ContainerDied","Data":"37c66de2ec51e9c681c41b5f5eef8c096539d001f5b048892716e31d27a6a68a"}
Jul 28 22:15:44.439066 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:44.438927    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gwv9l"
Jul 28 22:15:44.439066 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:44.439027    2563 scope.go:117] "RemoveContainer" containerID="715e5da2441edaaca227e6904cc70940ca32998380f19f81962ed2f6bde900ad"
Jul 28 22:15:44.450257 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:44.450227    2563 scope.go:117] "RemoveContainer" containerID="715e5da2441edaaca227e6904cc70940ca32998380f19f81962ed2f6bde900ad"
Jul 28 22:15:44.450849 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:15:44.450827    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"715e5da2441edaaca227e6904cc70940ca32998380f19f81962ed2f6bde900ad\": container with ID starting with 715e5da2441edaaca227e6904cc70940ca32998380f19f81962ed2f6bde900ad not found: ID does not exist" containerID="715e5da2441edaaca227e6904cc70940ca32998380f19f81962ed2f6bde900ad"
Jul 28 22:15:44.450936 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:44.450857    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"715e5da2441edaaca227e6904cc70940ca32998380f19f81962ed2f6bde900ad"} err="failed to get container status \"715e5da2441edaaca227e6904cc70940ca32998380f19f81962ed2f6bde900ad\": rpc error: code = NotFound desc = could not find container \"715e5da2441edaaca227e6904cc70940ca32998380f19f81962ed2f6bde900ad\": container with ID starting with 715e5da2441edaaca227e6904cc70940ca32998380f19f81962ed2f6bde900ad not found: ID does not exist"
Jul 28 22:15:44.464179 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:44.464132    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gwv9l"]
Jul 28 22:15:44.467276 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:44.467257    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gwv9l"]
Jul 28 22:15:44.542715 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:44.542688    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q6s9l"]
Jul 28 22:15:44.545396 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:15:44.545370    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad2a4a21_a0b5_4b90_ae3d_c29e89c26d01.slice/crio-8d6ceb86bb76c383d0a1dfb80189c8175ae5492f81235d8b92fca772631e3b22 WatchSource:0}: Error finding container 8d6ceb86bb76c383d0a1dfb80189c8175ae5492f81235d8b92fca772631e3b22: Status 404 returned error can't find the container with id 8d6ceb86bb76c383d0a1dfb80189c8175ae5492f81235d8b92fca772631e3b22
Jul 28 22:15:45.129594 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:45.129564    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aea07d6f-bf31-4ede-a50e-48a209883395" path="/var/lib/kubelet/pods/aea07d6f-bf31-4ede-a50e-48a209883395/volumes"
Jul 28 22:15:45.445019 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:45.444977    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q6s9l" event={"ID":"ad2a4a21-a0b5-4b90-ae3d-c29e89c26d01","Type":"ContainerStarted","Data":"94a5462239650c437ad5fe18f3cc20efdacb9436edefad8e07bb169030716d48"}
Jul 28 22:15:45.445194 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:45.445039    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q6s9l" event={"ID":"ad2a4a21-a0b5-4b90-ae3d-c29e89c26d01","Type":"ContainerStarted","Data":"8d6ceb86bb76c383d0a1dfb80189c8175ae5492f81235d8b92fca772631e3b22"}
Jul 28 22:15:45.445194 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:45.445066    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q6s9l"
Jul 28 22:15:45.468836 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:45.468783    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q6s9l" podStartSLOduration=1.468760069 podStartE2EDuration="1.468760069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:15:45.466446791 +0000 UTC m=+642.846584435" watchObservedRunningTime="2026-07-28 22:15:45.468760069 +0000 UTC m=+642.848897716"
Jul 28 22:15:56.452987 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:15:56.452949    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q6s9l"
Jul 28 22:16:14.602197 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:14.602164    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-72859"]
Jul 28 22:16:14.606272 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:14.606256    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-72859"
Jul 28 22:16:14.608870 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:14.608850    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 28 22:16:14.608870 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:14.608859    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cltnz\""
Jul 28 22:16:14.612820 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:14.612800    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-72859"]
Jul 28 22:16:14.701214 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:14.701181    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-72859"]
Jul 28 22:16:14.703027 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:14.703005    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6b99\" (UniqueName: \"kubernetes.io/projected/9ea0b1a7-b618-446c-af59-265dd7c0e95d-kube-api-access-s6b99\") pod \"limitador-limitador-64c94fd64f-72859\" (UID: \"9ea0b1a7-b618-446c-af59-265dd7c0e95d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-72859"
Jul 28 22:16:14.703091 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:14.703048    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9ea0b1a7-b618-446c-af59-265dd7c0e95d-config-file\") pod \"limitador-limitador-64c94fd64f-72859\" (UID: \"9ea0b1a7-b618-446c-af59-265dd7c0e95d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-72859"
Jul 28 22:16:14.803723 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:14.803684    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6b99\" (UniqueName: \"kubernetes.io/projected/9ea0b1a7-b618-446c-af59-265dd7c0e95d-kube-api-access-s6b99\") pod \"limitador-limitador-64c94fd64f-72859\" (UID: \"9ea0b1a7-b618-446c-af59-265dd7c0e95d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-72859"
Jul 28 22:16:14.803873 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:14.803734    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9ea0b1a7-b618-446c-af59-265dd7c0e95d-config-file\") pod \"limitador-limitador-64c94fd64f-72859\" (UID: \"9ea0b1a7-b618-446c-af59-265dd7c0e95d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-72859"
Jul 28 22:16:14.804391 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:14.804372    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9ea0b1a7-b618-446c-af59-265dd7c0e95d-config-file\") pod \"limitador-limitador-64c94fd64f-72859\" (UID: \"9ea0b1a7-b618-446c-af59-265dd7c0e95d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-72859"
Jul 28 22:16:14.813086 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:14.813059    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6b99\" (UniqueName: \"kubernetes.io/projected/9ea0b1a7-b618-446c-af59-265dd7c0e95d-kube-api-access-s6b99\") pod \"limitador-limitador-64c94fd64f-72859\" (UID: \"9ea0b1a7-b618-446c-af59-265dd7c0e95d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-72859"
Jul 28 22:16:14.917497 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:14.917408    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-72859"
Jul 28 22:16:15.056545 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:15.056451    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-72859"]
Jul 28 22:16:15.058982 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:16:15.058952    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ea0b1a7_b618_446c_af59_265dd7c0e95d.slice/crio-4b106d6af2ca22463693e990b3a912691f8387812407c666c3dece02c9ee425a WatchSource:0}: Error finding container 4b106d6af2ca22463693e990b3a912691f8387812407c666c3dece02c9ee425a: Status 404 returned error can't find the container with id 4b106d6af2ca22463693e990b3a912691f8387812407c666c3dece02c9ee425a
Jul 28 22:16:15.581324 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:15.581291    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-72859" event={"ID":"9ea0b1a7-b618-446c-af59-265dd7c0e95d","Type":"ContainerStarted","Data":"4b106d6af2ca22463693e990b3a912691f8387812407c666c3dece02c9ee425a"}
Jul 28 22:16:18.599009 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:18.598968    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-72859" event={"ID":"9ea0b1a7-b618-446c-af59-265dd7c0e95d","Type":"ContainerStarted","Data":"d9fa672238e79bdfb83a5402e665cecdb22fd6c9771e0782bfb1da72aa504bfb"}
Jul 28 22:16:18.599465 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:18.599094    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-72859"
Jul 28 22:16:18.616491 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:18.616449    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-72859" podStartSLOduration=1.998591877 podStartE2EDuration="4.616436503s" totalImagesPullingTime="2.617844626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:16:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:16:18.615290531 +0000 UTC m=+675.995428176" watchObservedRunningTime="2026-07-28 22:16:18.616436503 +0000 UTC m=+675.996574147"
Jul 28 22:16:29.053457 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:29.053422    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-72859"]
Jul 28 22:16:29.053959 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:29.053609    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-72859" podUID="9ea0b1a7-b618-446c-af59-265dd7c0e95d" containerName="limitador" containerID="cri-o://d9fa672238e79bdfb83a5402e665cecdb22fd6c9771e0782bfb1da72aa504bfb" gracePeriod=30
Jul 28 22:16:29.054336 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:29.054318    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-72859"
Jul 28 22:16:29.605778 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:29.605756    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-72859"
Jul 28 22:16:29.646413 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:29.646326    2563 generic.go:356] "Generic (PLEG): container finished" podID="9ea0b1a7-b618-446c-af59-265dd7c0e95d" containerID="d9fa672238e79bdfb83a5402e665cecdb22fd6c9771e0782bfb1da72aa504bfb" exitCode=0
Jul 28 22:16:29.646413 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:29.646381    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-72859" event={"ID":"9ea0b1a7-b618-446c-af59-265dd7c0e95d","Type":"ContainerDied","Data":"d9fa672238e79bdfb83a5402e665cecdb22fd6c9771e0782bfb1da72aa504bfb"}
Jul 28 22:16:29.646413 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:29.646399    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-72859"
Jul 28 22:16:29.646655 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:29.646435    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-72859" event={"ID":"9ea0b1a7-b618-446c-af59-265dd7c0e95d","Type":"ContainerDied","Data":"4b106d6af2ca22463693e990b3a912691f8387812407c666c3dece02c9ee425a"}
Jul 28 22:16:29.646655 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:29.646462    2563 scope.go:117] "RemoveContainer" containerID="d9fa672238e79bdfb83a5402e665cecdb22fd6c9771e0782bfb1da72aa504bfb"
Jul 28 22:16:29.654523 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:29.654505    2563 scope.go:117] "RemoveContainer" containerID="d9fa672238e79bdfb83a5402e665cecdb22fd6c9771e0782bfb1da72aa504bfb"
Jul 28 22:16:29.654812 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:16:29.654794    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9fa672238e79bdfb83a5402e665cecdb22fd6c9771e0782bfb1da72aa504bfb\": container with ID starting with d9fa672238e79bdfb83a5402e665cecdb22fd6c9771e0782bfb1da72aa504bfb not found: ID does not exist" containerID="d9fa672238e79bdfb83a5402e665cecdb22fd6c9771e0782bfb1da72aa504bfb"
Jul 28 22:16:29.654879 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:29.654820    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9fa672238e79bdfb83a5402e665cecdb22fd6c9771e0782bfb1da72aa504bfb"} err="failed to get container status \"d9fa672238e79bdfb83a5402e665cecdb22fd6c9771e0782bfb1da72aa504bfb\": rpc error: code = NotFound desc = could not find container \"d9fa672238e79bdfb83a5402e665cecdb22fd6c9771e0782bfb1da72aa504bfb\": container with ID starting with d9fa672238e79bdfb83a5402e665cecdb22fd6c9771e0782bfb1da72aa504bfb not found: ID does not exist"
Jul 28 22:16:29.725739 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:29.725702    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6b99\" (UniqueName: \"kubernetes.io/projected/9ea0b1a7-b618-446c-af59-265dd7c0e95d-kube-api-access-s6b99\") pod \"9ea0b1a7-b618-446c-af59-265dd7c0e95d\" (UID: \"9ea0b1a7-b618-446c-af59-265dd7c0e95d\") "
Jul 28 22:16:29.725868 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:29.725787    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9ea0b1a7-b618-446c-af59-265dd7c0e95d-config-file\") pod \"9ea0b1a7-b618-446c-af59-265dd7c0e95d\" (UID: \"9ea0b1a7-b618-446c-af59-265dd7c0e95d\") "
Jul 28 22:16:29.726129 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:29.726106    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ea0b1a7-b618-446c-af59-265dd7c0e95d-config-file" (OuterVolumeSpecName: "config-file") pod "9ea0b1a7-b618-446c-af59-265dd7c0e95d" (UID: "9ea0b1a7-b618-446c-af59-265dd7c0e95d"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:16:29.727623 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:29.727601    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ea0b1a7-b618-446c-af59-265dd7c0e95d-kube-api-access-s6b99" (OuterVolumeSpecName: "kube-api-access-s6b99") pod "9ea0b1a7-b618-446c-af59-265dd7c0e95d" (UID: "9ea0b1a7-b618-446c-af59-265dd7c0e95d"). InnerVolumeSpecName "kube-api-access-s6b99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:16:29.826765 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:29.826732    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6b99\" (UniqueName: \"kubernetes.io/projected/9ea0b1a7-b618-446c-af59-265dd7c0e95d-kube-api-access-s6b99\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:16:29.826765 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:29.826760    2563 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9ea0b1a7-b618-446c-af59-265dd7c0e95d-config-file\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:16:29.968015 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:29.967984    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-72859"]
Jul 28 22:16:29.971877 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:29.971855    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-72859"]
Jul 28 22:16:31.129450 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:31.129417    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ea0b1a7-b618-446c-af59-265dd7c0e95d" path="/var/lib/kubelet/pods/9ea0b1a7-b618-446c-af59-265dd7c0e95d/volumes"
Jul 28 22:16:33.936511 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:33.936479    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-h7cc5"]
Jul 28 22:16:33.936883 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:33.936854    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ea0b1a7-b618-446c-af59-265dd7c0e95d" containerName="limitador"
Jul 28 22:16:33.936883 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:33.936874    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ea0b1a7-b618-446c-af59-265dd7c0e95d" containerName="limitador"
Jul 28 22:16:33.936970 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:33.936953    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ea0b1a7-b618-446c-af59-265dd7c0e95d" containerName="limitador"
Jul 28 22:16:33.941402 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:33.941383    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-h7cc5"
Jul 28 22:16:33.943969 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:33.943947    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 28 22:16:33.944114 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:33.944096    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-tnwqz\""
Jul 28 22:16:33.945128 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:33.945111    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 22:16:33.945128 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:33.945123    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 22:16:33.948955 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:33.948933    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-h7cc5"]
Jul 28 22:16:34.058950 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:34.058915    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/07f92c4d-16a3-41de-8d51-ecfa6e5a32b5-data\") pod \"postgres-69fb4594cf-h7cc5\" (UID: \"07f92c4d-16a3-41de-8d51-ecfa6e5a32b5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-h7cc5"
Jul 28 22:16:34.059101 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:34.059002    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc4nq\" (UniqueName: \"kubernetes.io/projected/07f92c4d-16a3-41de-8d51-ecfa6e5a32b5-kube-api-access-fc4nq\") pod \"postgres-69fb4594cf-h7cc5\" (UID: \"07f92c4d-16a3-41de-8d51-ecfa6e5a32b5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-h7cc5"
Jul 28 22:16:34.159826 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:34.159786    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/07f92c4d-16a3-41de-8d51-ecfa6e5a32b5-data\") pod \"postgres-69fb4594cf-h7cc5\" (UID: \"07f92c4d-16a3-41de-8d51-ecfa6e5a32b5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-h7cc5"
Jul 28 22:16:34.160017 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:34.159876    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc4nq\" (UniqueName: \"kubernetes.io/projected/07f92c4d-16a3-41de-8d51-ecfa6e5a32b5-kube-api-access-fc4nq\") pod \"postgres-69fb4594cf-h7cc5\" (UID: \"07f92c4d-16a3-41de-8d51-ecfa6e5a32b5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-h7cc5"
Jul 28 22:16:34.160169 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:34.160147    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/07f92c4d-16a3-41de-8d51-ecfa6e5a32b5-data\") pod \"postgres-69fb4594cf-h7cc5\" (UID: \"07f92c4d-16a3-41de-8d51-ecfa6e5a32b5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-h7cc5"
Jul 28 22:16:34.168520 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:34.168494    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc4nq\" (UniqueName: \"kubernetes.io/projected/07f92c4d-16a3-41de-8d51-ecfa6e5a32b5-kube-api-access-fc4nq\") pod \"postgres-69fb4594cf-h7cc5\" (UID: \"07f92c4d-16a3-41de-8d51-ecfa6e5a32b5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-h7cc5"
Jul 28 22:16:34.250967 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:34.250891    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-h7cc5"
Jul 28 22:16:34.378757 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:34.378728    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-h7cc5"]
Jul 28 22:16:34.381990 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:16:34.381964    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f92c4d_16a3_41de_8d51_ecfa6e5a32b5.slice/crio-a7b420fada58348dc4450949bdfbafb37557517b9ad1bb311405a586b67db005 WatchSource:0}: Error finding container a7b420fada58348dc4450949bdfbafb37557517b9ad1bb311405a586b67db005: Status 404 returned error can't find the container with id a7b420fada58348dc4450949bdfbafb37557517b9ad1bb311405a586b67db005
Jul 28 22:16:34.383633 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:34.383614    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 22:16:34.674705 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:34.674616    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-h7cc5" event={"ID":"07f92c4d-16a3-41de-8d51-ecfa6e5a32b5","Type":"ContainerStarted","Data":"a7b420fada58348dc4450949bdfbafb37557517b9ad1bb311405a586b67db005"}
Jul 28 22:16:39.702578 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:39.702543    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-h7cc5" event={"ID":"07f92c4d-16a3-41de-8d51-ecfa6e5a32b5","Type":"ContainerStarted","Data":"222cf192cf2b9cfd96bb663117dd15166cce9f664427ef07b77bc0ce580422b3"}
Jul 28 22:16:39.702961 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:39.702653    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-h7cc5"
Jul 28 22:16:39.718922 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:39.718876    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-h7cc5" podStartSLOduration=1.867764061 podStartE2EDuration="6.718860153s" totalImagesPullingTime="4.851096092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:16:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:16:39.717325662 +0000 UTC m=+697.097463298" watchObservedRunningTime="2026-07-28 22:16:39.718860153 +0000 UTC m=+697.098997797"
Jul 28 22:16:45.735253 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:45.735220    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-h7cc5"
Jul 28 22:16:46.548594 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:46.548561    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7dbd8867fc-kn4zx"]
Jul 28 22:16:46.554571 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:46.554549    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7dbd8867fc-kn4zx"
Jul 28 22:16:46.557444 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:46.557420    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-pb829\""
Jul 28 22:16:46.558743 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:46.558722    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7dbd8867fc-kn4zx"]
Jul 28 22:16:46.675515 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:46.675480    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2tmk\" (UniqueName: \"kubernetes.io/projected/471d5751-f19c-49d9-a539-950df39ab231-kube-api-access-s2tmk\") pod \"authorino-7dbd8867fc-kn4zx\" (UID: \"471d5751-f19c-49d9-a539-950df39ab231\") " pod="kuadrant-system/authorino-7dbd8867fc-kn4zx"
Jul 28 22:16:46.724574 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:46.724535    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7dbd8867fc-kn4zx"]
Jul 28 22:16:46.724817 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:16:46.724793    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-s2tmk], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-7dbd8867fc-kn4zx" podUID="471d5751-f19c-49d9-a539-950df39ab231"
Jul 28 22:16:46.733703 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:46.733680    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7dbd8867fc-kn4zx"
Jul 28 22:16:46.740895 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:46.740876    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7dbd8867fc-kn4zx"
Jul 28 22:16:46.776694 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:46.776671    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2tmk\" (UniqueName: \"kubernetes.io/projected/471d5751-f19c-49d9-a539-950df39ab231-kube-api-access-s2tmk\") pod \"authorino-7dbd8867fc-kn4zx\" (UID: \"471d5751-f19c-49d9-a539-950df39ab231\") " pod="kuadrant-system/authorino-7dbd8867fc-kn4zx"
Jul 28 22:16:46.785732 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:46.785707    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2tmk\" (UniqueName: \"kubernetes.io/projected/471d5751-f19c-49d9-a539-950df39ab231-kube-api-access-s2tmk\") pod \"authorino-7dbd8867fc-kn4zx\" (UID: \"471d5751-f19c-49d9-a539-950df39ab231\") " pod="kuadrant-system/authorino-7dbd8867fc-kn4zx"
Jul 28 22:16:46.877724 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:46.877657    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2tmk\" (UniqueName: \"kubernetes.io/projected/471d5751-f19c-49d9-a539-950df39ab231-kube-api-access-s2tmk\") pod \"471d5751-f19c-49d9-a539-950df39ab231\" (UID: \"471d5751-f19c-49d9-a539-950df39ab231\") "
Jul 28 22:16:46.879670 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:46.879648    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/471d5751-f19c-49d9-a539-950df39ab231-kube-api-access-s2tmk" (OuterVolumeSpecName: "kube-api-access-s2tmk") pod "471d5751-f19c-49d9-a539-950df39ab231" (UID: "471d5751-f19c-49d9-a539-950df39ab231"). InnerVolumeSpecName "kube-api-access-s2tmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:16:46.978979 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:46.978950    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2tmk\" (UniqueName: \"kubernetes.io/projected/471d5751-f19c-49d9-a539-950df39ab231-kube-api-access-s2tmk\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:16:47.738443 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:47.738412    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7dbd8867fc-kn4zx"
Jul 28 22:16:47.763064 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:47.763038    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7dbd8867fc-kn4zx"]
Jul 28 22:16:47.765045 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:47.765020    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7dbd8867fc-kn4zx"]
Jul 28 22:16:49.129343 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:49.129304    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="471d5751-f19c-49d9-a539-950df39ab231" path="/var/lib/kubelet/pods/471d5751-f19c-49d9-a539-950df39ab231/volumes"
Jul 28 22:16:53.112176 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:53.112145    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-598548b8fd-h5rdw"]
Jul 28 22:16:53.312553 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:53.312525    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-598548b8fd-h5rdw"
Jul 28 22:16:53.315587 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:53.315560    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 28 22:16:53.315726 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:53.315599    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 28 22:16:53.315726 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:53.315664    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mtsxn\""
Jul 28 22:16:53.320751 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:53.320729    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-598548b8fd-h5rdw"]
Jul 28 22:16:53.320845 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:53.320769    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8687496d99-pzsps"]
Jul 28 22:16:53.338393 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:53.338370    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8687496d99-pzsps"]
Jul 28 22:16:53.338480 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:53.338423    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8687496d99-pzsps"
Jul 28 22:16:53.431586 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:53.431520    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/856899f8-c889-4c7b-acca-7b773b624420-webhook-cert\") pod \"maas-controller-8687496d99-pzsps\" (UID: \"856899f8-c889-4c7b-acca-7b773b624420\") " pod="opendatahub/maas-controller-8687496d99-pzsps"
Jul 28 22:16:53.431718 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:53.431592    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qxp2\" (UniqueName: \"kubernetes.io/projected/424eb211-6eae-467d-b8af-73b6409a8125-kube-api-access-9qxp2\") pod \"maas-controller-598548b8fd-h5rdw\" (UID: \"424eb211-6eae-467d-b8af-73b6409a8125\") " pod="opendatahub/maas-controller-598548b8fd-h5rdw"
Jul 28 22:16:53.431718 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:53.431613    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/424eb211-6eae-467d-b8af-73b6409a8125-webhook-cert\") pod \"maas-controller-598548b8fd-h5rdw\" (UID: \"424eb211-6eae-467d-b8af-73b6409a8125\") " pod="opendatahub/maas-controller-598548b8fd-h5rdw"
Jul 28 22:16:53.431718 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:53.431630    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sjvk\" (UniqueName: \"kubernetes.io/projected/856899f8-c889-4c7b-acca-7b773b624420-kube-api-access-7sjvk\") pod \"maas-controller-8687496d99-pzsps\" (UID: \"856899f8-c889-4c7b-acca-7b773b624420\") " pod="opendatahub/maas-controller-8687496d99-pzsps"
Jul 28 22:16:53.532751 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:53.532724    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/424eb211-6eae-467d-b8af-73b6409a8125-webhook-cert\") pod \"maas-controller-598548b8fd-h5rdw\" (UID: \"424eb211-6eae-467d-b8af-73b6409a8125\") " pod="opendatahub/maas-controller-598548b8fd-h5rdw"
Jul 28 22:16:53.532893 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:53.532754    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sjvk\" (UniqueName: \"kubernetes.io/projected/856899f8-c889-4c7b-acca-7b773b624420-kube-api-access-7sjvk\") pod \"maas-controller-8687496d99-pzsps\" (UID: \"856899f8-c889-4c7b-acca-7b773b624420\") " pod="opendatahub/maas-controller-8687496d99-pzsps"
Jul 28 22:16:53.532893 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:53.532785    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/856899f8-c889-4c7b-acca-7b773b624420-webhook-cert\") pod \"maas-controller-8687496d99-pzsps\" (UID: \"856899f8-c889-4c7b-acca-7b773b624420\") " pod="opendatahub/maas-controller-8687496d99-pzsps"
Jul 28 22:16:53.532893 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:53.532835    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qxp2\" (UniqueName: \"kubernetes.io/projected/424eb211-6eae-467d-b8af-73b6409a8125-kube-api-access-9qxp2\") pod \"maas-controller-598548b8fd-h5rdw\" (UID: \"424eb211-6eae-467d-b8af-73b6409a8125\") " pod="opendatahub/maas-controller-598548b8fd-h5rdw"
Jul 28 22:16:53.532893 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:16:53.532871    2563 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 28 22:16:53.533081 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:16:53.532928    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/424eb211-6eae-467d-b8af-73b6409a8125-webhook-cert podName:424eb211-6eae-467d-b8af-73b6409a8125 nodeName:}" failed. No retries permitted until 2026-07-28 22:16:54.032912524 +0000 UTC m=+711.413050146 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/424eb211-6eae-467d-b8af-73b6409a8125-webhook-cert") pod "maas-controller-598548b8fd-h5rdw" (UID: "424eb211-6eae-467d-b8af-73b6409a8125") : secret "maas-controller-webhook-cert" not found
Jul 28 22:16:53.533081 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:16:53.532967    2563 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 28 22:16:53.533081 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:16:53.533033    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/856899f8-c889-4c7b-acca-7b773b624420-webhook-cert podName:856899f8-c889-4c7b-acca-7b773b624420 nodeName:}" failed. No retries permitted until 2026-07-28 22:16:54.033016325 +0000 UTC m=+711.413153963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/856899f8-c889-4c7b-acca-7b773b624420-webhook-cert") pod "maas-controller-8687496d99-pzsps" (UID: "856899f8-c889-4c7b-acca-7b773b624420") : secret "maas-controller-webhook-cert" not found
Jul 28 22:16:53.542060 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:53.542032    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sjvk\" (UniqueName: \"kubernetes.io/projected/856899f8-c889-4c7b-acca-7b773b624420-kube-api-access-7sjvk\") pod \"maas-controller-8687496d99-pzsps\" (UID: \"856899f8-c889-4c7b-acca-7b773b624420\") " pod="opendatahub/maas-controller-8687496d99-pzsps"
Jul 28 22:16:53.542163 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:53.542077    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qxp2\" (UniqueName: \"kubernetes.io/projected/424eb211-6eae-467d-b8af-73b6409a8125-kube-api-access-9qxp2\") pod \"maas-controller-598548b8fd-h5rdw\" (UID: \"424eb211-6eae-467d-b8af-73b6409a8125\") " pod="opendatahub/maas-controller-598548b8fd-h5rdw"
Jul 28 22:16:54.037817 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:54.037775    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/856899f8-c889-4c7b-acca-7b773b624420-webhook-cert\") pod \"maas-controller-8687496d99-pzsps\" (UID: \"856899f8-c889-4c7b-acca-7b773b624420\") " pod="opendatahub/maas-controller-8687496d99-pzsps"
Jul 28 22:16:54.038035 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:54.037882    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/424eb211-6eae-467d-b8af-73b6409a8125-webhook-cert\") pod \"maas-controller-598548b8fd-h5rdw\" (UID: \"424eb211-6eae-467d-b8af-73b6409a8125\") " pod="opendatahub/maas-controller-598548b8fd-h5rdw"
Jul 28 22:16:54.040131 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:54.040101    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/856899f8-c889-4c7b-acca-7b773b624420-webhook-cert\") pod \"maas-controller-8687496d99-pzsps\" (UID: \"856899f8-c889-4c7b-acca-7b773b624420\") " pod="opendatahub/maas-controller-8687496d99-pzsps"
Jul 28 22:16:54.040252 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:54.040197    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/424eb211-6eae-467d-b8af-73b6409a8125-webhook-cert\") pod \"maas-controller-598548b8fd-h5rdw\" (UID: \"424eb211-6eae-467d-b8af-73b6409a8125\") " pod="opendatahub/maas-controller-598548b8fd-h5rdw"
Jul 28 22:16:54.223501 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:54.223464    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-598548b8fd-h5rdw"
Jul 28 22:16:54.248130 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:54.248102    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8687496d99-pzsps"
Jul 28 22:16:54.375566 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:54.375391    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-598548b8fd-h5rdw"]
Jul 28 22:16:54.377611 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:16:54.377585    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod424eb211_6eae_467d_b8af_73b6409a8125.slice/crio-d1d886ef1a5e55be28d33e6d7e4dae06844ae9b0b565e0160cf2f4d8a2457a85 WatchSource:0}: Error finding container d1d886ef1a5e55be28d33e6d7e4dae06844ae9b0b565e0160cf2f4d8a2457a85: Status 404 returned error can't find the container with id d1d886ef1a5e55be28d33e6d7e4dae06844ae9b0b565e0160cf2f4d8a2457a85
Jul 28 22:16:54.396273 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:54.396209    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8687496d99-pzsps"]
Jul 28 22:16:54.398595 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:16:54.398572    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod856899f8_c889_4c7b_acca_7b773b624420.slice/crio-0f854fa0aec70a5d99c8700c7e9f939a0b044fdcb3727bd5a42e4fd51d120cbe WatchSource:0}: Error finding container 0f854fa0aec70a5d99c8700c7e9f939a0b044fdcb3727bd5a42e4fd51d120cbe: Status 404 returned error can't find the container with id 0f854fa0aec70a5d99c8700c7e9f939a0b044fdcb3727bd5a42e4fd51d120cbe
Jul 28 22:16:54.769427 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:54.769389    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8687496d99-pzsps" event={"ID":"856899f8-c889-4c7b-acca-7b773b624420","Type":"ContainerStarted","Data":"0f854fa0aec70a5d99c8700c7e9f939a0b044fdcb3727bd5a42e4fd51d120cbe"}
Jul 28 22:16:54.770455 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:54.770421    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-598548b8fd-h5rdw" event={"ID":"424eb211-6eae-467d-b8af-73b6409a8125","Type":"ContainerStarted","Data":"d1d886ef1a5e55be28d33e6d7e4dae06844ae9b0b565e0160cf2f4d8a2457a85"}
Jul 28 22:16:57.788184 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:57.788147    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-598548b8fd-h5rdw" event={"ID":"424eb211-6eae-467d-b8af-73b6409a8125","Type":"ContainerStarted","Data":"72d46e3beb9f82b183ec6b91a99aed1689f1d8fe7af405a8d36ffb6bbeab9898"}
Jul 28 22:16:57.788650 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:57.788212    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-598548b8fd-h5rdw"
Jul 28 22:16:57.789538 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:57.789514    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8687496d99-pzsps" event={"ID":"856899f8-c889-4c7b-acca-7b773b624420","Type":"ContainerStarted","Data":"38ea543c5da771e80d648ed25edf6d441c2280bc190d00d6793c3d93d80e2da6"}
Jul 28 22:16:57.789636 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:57.789559    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8687496d99-pzsps"
Jul 28 22:16:57.806633 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:57.806585    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-598548b8fd-h5rdw" podStartSLOduration=2.326198098 podStartE2EDuration="4.806569258s" totalImagesPullingTime="2.48037116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:16:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:16:57.805127991 +0000 UTC m=+715.185265636" watchObservedRunningTime="2026-07-28 22:16:57.806569258 +0000 UTC m=+715.186706901"
Jul 28 22:16:57.822908 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:16:57.822857    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-8687496d99-pzsps" podStartSLOduration=2.372701004 podStartE2EDuration="4.822841574s" totalImagesPullingTime="2.45014057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:16:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:16:57.821839434 +0000 UTC m=+715.201977105" watchObservedRunningTime="2026-07-28 22:16:57.822841574 +0000 UTC m=+715.202979218"
Jul 28 22:17:08.796598 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:08.796568    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8687496d99-pzsps"
Jul 28 22:17:08.797046 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:08.796639    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-598548b8fd-h5rdw"
Jul 28 22:17:08.862299 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:08.862267    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-598548b8fd-h5rdw"]
Jul 28 22:17:08.862503 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:08.862482    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-598548b8fd-h5rdw" podUID="424eb211-6eae-467d-b8af-73b6409a8125" containerName="manager" containerID="cri-o://72d46e3beb9f82b183ec6b91a99aed1689f1d8fe7af405a8d36ffb6bbeab9898" gracePeriod=30
Jul 28 22:17:09.112387 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:09.112339    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-598548b8fd-h5rdw"
Jul 28 22:17:09.168866 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:09.168834    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qxp2\" (UniqueName: \"kubernetes.io/projected/424eb211-6eae-467d-b8af-73b6409a8125-kube-api-access-9qxp2\") pod \"424eb211-6eae-467d-b8af-73b6409a8125\" (UID: \"424eb211-6eae-467d-b8af-73b6409a8125\") "
Jul 28 22:17:09.169007 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:09.168891    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/424eb211-6eae-467d-b8af-73b6409a8125-webhook-cert\") pod \"424eb211-6eae-467d-b8af-73b6409a8125\" (UID: \"424eb211-6eae-467d-b8af-73b6409a8125\") "
Jul 28 22:17:09.170943 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:09.170912    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/424eb211-6eae-467d-b8af-73b6409a8125-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "424eb211-6eae-467d-b8af-73b6409a8125" (UID: "424eb211-6eae-467d-b8af-73b6409a8125"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:17:09.171070 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:09.170982    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/424eb211-6eae-467d-b8af-73b6409a8125-kube-api-access-9qxp2" (OuterVolumeSpecName: "kube-api-access-9qxp2") pod "424eb211-6eae-467d-b8af-73b6409a8125" (UID: "424eb211-6eae-467d-b8af-73b6409a8125"). InnerVolumeSpecName "kube-api-access-9qxp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:17:09.269555 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:09.269522    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qxp2\" (UniqueName: \"kubernetes.io/projected/424eb211-6eae-467d-b8af-73b6409a8125-kube-api-access-9qxp2\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:17:09.269555 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:09.269552    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/424eb211-6eae-467d-b8af-73b6409a8125-webhook-cert\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:17:09.850054 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:09.850015    2563 generic.go:356] "Generic (PLEG): container finished" podID="424eb211-6eae-467d-b8af-73b6409a8125" containerID="72d46e3beb9f82b183ec6b91a99aed1689f1d8fe7af405a8d36ffb6bbeab9898" exitCode=0
Jul 28 22:17:09.850514 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:09.850074    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-598548b8fd-h5rdw"
Jul 28 22:17:09.850514 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:09.850103    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-598548b8fd-h5rdw" event={"ID":"424eb211-6eae-467d-b8af-73b6409a8125","Type":"ContainerDied","Data":"72d46e3beb9f82b183ec6b91a99aed1689f1d8fe7af405a8d36ffb6bbeab9898"}
Jul 28 22:17:09.850514 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:09.850156    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-598548b8fd-h5rdw" event={"ID":"424eb211-6eae-467d-b8af-73b6409a8125","Type":"ContainerDied","Data":"d1d886ef1a5e55be28d33e6d7e4dae06844ae9b0b565e0160cf2f4d8a2457a85"}
Jul 28 22:17:09.850514 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:09.850180    2563 scope.go:117] "RemoveContainer" containerID="72d46e3beb9f82b183ec6b91a99aed1689f1d8fe7af405a8d36ffb6bbeab9898"
Jul 28 22:17:09.859161 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:09.859142    2563 scope.go:117] "RemoveContainer" containerID="72d46e3beb9f82b183ec6b91a99aed1689f1d8fe7af405a8d36ffb6bbeab9898"
Jul 28 22:17:09.859454 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:17:09.859434    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72d46e3beb9f82b183ec6b91a99aed1689f1d8fe7af405a8d36ffb6bbeab9898\": container with ID starting with 72d46e3beb9f82b183ec6b91a99aed1689f1d8fe7af405a8d36ffb6bbeab9898 not found: ID does not exist" containerID="72d46e3beb9f82b183ec6b91a99aed1689f1d8fe7af405a8d36ffb6bbeab9898"
Jul 28 22:17:09.859508 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:09.859462    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72d46e3beb9f82b183ec6b91a99aed1689f1d8fe7af405a8d36ffb6bbeab9898"} err="failed to get container status \"72d46e3beb9f82b183ec6b91a99aed1689f1d8fe7af405a8d36ffb6bbeab9898\": rpc error: code = NotFound desc = could not find container \"72d46e3beb9f82b183ec6b91a99aed1689f1d8fe7af405a8d36ffb6bbeab9898\": container with ID starting with 72d46e3beb9f82b183ec6b91a99aed1689f1d8fe7af405a8d36ffb6bbeab9898 not found: ID does not exist"
Jul 28 22:17:09.873513 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:09.873486    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-598548b8fd-h5rdw"]
Jul 28 22:17:09.879444 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:09.879418    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-598548b8fd-h5rdw"]
Jul 28 22:17:11.130198 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:11.130163    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="424eb211-6eae-467d-b8af-73b6409a8125" path="/var/lib/kubelet/pods/424eb211-6eae-467d-b8af-73b6409a8125/volumes"
Jul 28 22:17:29.774674 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:29.774638    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-fcfcd7d96-cpvhs"]
Jul 28 22:17:29.775086 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:29.774949    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="424eb211-6eae-467d-b8af-73b6409a8125" containerName="manager"
Jul 28 22:17:29.775086 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:29.774960    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="424eb211-6eae-467d-b8af-73b6409a8125" containerName="manager"
Jul 28 22:17:29.775086 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:29.775012    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="424eb211-6eae-467d-b8af-73b6409a8125" containerName="manager"
Jul 28 22:17:29.777831 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:29.777815    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-fcfcd7d96-cpvhs"
Jul 28 22:17:29.781894 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:29.781873    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 28 22:17:29.782015 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:29.781875    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-46ntd\""
Jul 28 22:17:29.789433 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:29.789406    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-fcfcd7d96-cpvhs"]
Jul 28 22:17:29.844778 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:29.844746    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b67e040-3901-4cf0-bd2d-3224954ba70e-maas-api-tls\") pod \"maas-api-fcfcd7d96-cpvhs\" (UID: \"1b67e040-3901-4cf0-bd2d-3224954ba70e\") " pod="odh-ai-gateway-infra/maas-api-fcfcd7d96-cpvhs"
Jul 28 22:17:29.844778 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:29.844781    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njggq\" (UniqueName: \"kubernetes.io/projected/1b67e040-3901-4cf0-bd2d-3224954ba70e-kube-api-access-njggq\") pod \"maas-api-fcfcd7d96-cpvhs\" (UID: \"1b67e040-3901-4cf0-bd2d-3224954ba70e\") " pod="odh-ai-gateway-infra/maas-api-fcfcd7d96-cpvhs"
Jul 28 22:17:29.844995 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:29.844818    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b67e040-3901-4cf0-bd2d-3224954ba70e-openshift-service-ca\") pod \"maas-api-fcfcd7d96-cpvhs\" (UID: \"1b67e040-3901-4cf0-bd2d-3224954ba70e\") " pod="odh-ai-gateway-infra/maas-api-fcfcd7d96-cpvhs"
Jul 28 22:17:29.945317 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:29.945280    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b67e040-3901-4cf0-bd2d-3224954ba70e-openshift-service-ca\") pod \"maas-api-fcfcd7d96-cpvhs\" (UID: \"1b67e040-3901-4cf0-bd2d-3224954ba70e\") " pod="odh-ai-gateway-infra/maas-api-fcfcd7d96-cpvhs"
Jul 28 22:17:29.945532 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:29.945375    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b67e040-3901-4cf0-bd2d-3224954ba70e-maas-api-tls\") pod \"maas-api-fcfcd7d96-cpvhs\" (UID: \"1b67e040-3901-4cf0-bd2d-3224954ba70e\") " pod="odh-ai-gateway-infra/maas-api-fcfcd7d96-cpvhs"
Jul 28 22:17:29.945532 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:29.945402    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njggq\" (UniqueName: \"kubernetes.io/projected/1b67e040-3901-4cf0-bd2d-3224954ba70e-kube-api-access-njggq\") pod \"maas-api-fcfcd7d96-cpvhs\" (UID: \"1b67e040-3901-4cf0-bd2d-3224954ba70e\") " pod="odh-ai-gateway-infra/maas-api-fcfcd7d96-cpvhs"
Jul 28 22:17:29.945996 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:29.945974    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b67e040-3901-4cf0-bd2d-3224954ba70e-openshift-service-ca\") pod \"maas-api-fcfcd7d96-cpvhs\" (UID: \"1b67e040-3901-4cf0-bd2d-3224954ba70e\") " pod="odh-ai-gateway-infra/maas-api-fcfcd7d96-cpvhs"
Jul 28 22:17:29.947790 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:29.947762    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b67e040-3901-4cf0-bd2d-3224954ba70e-maas-api-tls\") pod \"maas-api-fcfcd7d96-cpvhs\" (UID: \"1b67e040-3901-4cf0-bd2d-3224954ba70e\") " pod="odh-ai-gateway-infra/maas-api-fcfcd7d96-cpvhs"
Jul 28 22:17:29.957550 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:29.957531    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njggq\" (UniqueName: \"kubernetes.io/projected/1b67e040-3901-4cf0-bd2d-3224954ba70e-kube-api-access-njggq\") pod \"maas-api-fcfcd7d96-cpvhs\" (UID: \"1b67e040-3901-4cf0-bd2d-3224954ba70e\") " pod="odh-ai-gateway-infra/maas-api-fcfcd7d96-cpvhs"
Jul 28 22:17:30.087775 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:30.087699    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-fcfcd7d96-cpvhs"
Jul 28 22:17:30.231115 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:30.231091    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-fcfcd7d96-cpvhs"]
Jul 28 22:17:30.233629 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:17:30.233592    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b67e040_3901_4cf0_bd2d_3224954ba70e.slice/crio-aa47faf0e3de2daab27a2a6ee43a672fa839b3c1d1703240b91cec24c584ecf2 WatchSource:0}: Error finding container aa47faf0e3de2daab27a2a6ee43a672fa839b3c1d1703240b91cec24c584ecf2: Status 404 returned error can't find the container with id aa47faf0e3de2daab27a2a6ee43a672fa839b3c1d1703240b91cec24c584ecf2
Jul 28 22:17:30.263144 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:30.263118    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-2zd9d"]
Jul 28 22:17:30.267447 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:30.267424    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2zd9d"
Jul 28 22:17:30.269867 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:30.269845    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 28 22:17:30.269981 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:30.269934    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-z2xh7\""
Jul 28 22:17:30.277024 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:30.277001    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-2zd9d"]
Jul 28 22:17:30.349439 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:30.349363    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wjpm\" (UniqueName: \"kubernetes.io/projected/b164ba31-adb2-42c8-9704-d398ba2650eb-kube-api-access-8wjpm\") pod \"payload-pre-processing-687dbb6cf6-2zd9d\" (UID: \"b164ba31-adb2-42c8-9704-d398ba2650eb\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2zd9d"
Jul 28 22:17:30.349439 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:30.349428    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b164ba31-adb2-42c8-9704-d398ba2650eb-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-2zd9d\" (UID: \"b164ba31-adb2-42c8-9704-d398ba2650eb\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2zd9d"
Jul 28 22:17:30.450925 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:30.450879    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wjpm\" (UniqueName: \"kubernetes.io/projected/b164ba31-adb2-42c8-9704-d398ba2650eb-kube-api-access-8wjpm\") pod \"payload-pre-processing-687dbb6cf6-2zd9d\" (UID: \"b164ba31-adb2-42c8-9704-d398ba2650eb\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2zd9d"
Jul 28 22:17:30.451130 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:30.450971    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b164ba31-adb2-42c8-9704-d398ba2650eb-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-2zd9d\" (UID: \"b164ba31-adb2-42c8-9704-d398ba2650eb\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2zd9d"
Jul 28 22:17:30.451696 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:30.451675    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b164ba31-adb2-42c8-9704-d398ba2650eb-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-2zd9d\" (UID: \"b164ba31-adb2-42c8-9704-d398ba2650eb\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2zd9d"
Jul 28 22:17:30.465851 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:30.465820    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wjpm\" (UniqueName: \"kubernetes.io/projected/b164ba31-adb2-42c8-9704-d398ba2650eb-kube-api-access-8wjpm\") pod \"payload-pre-processing-687dbb6cf6-2zd9d\" (UID: \"b164ba31-adb2-42c8-9704-d398ba2650eb\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2zd9d"
Jul 28 22:17:30.579781 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:30.579738    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2zd9d"
Jul 28 22:17:30.721408 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:30.721384    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-2zd9d"]
Jul 28 22:17:30.723704 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:17:30.723672    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb164ba31_adb2_42c8_9704_d398ba2650eb.slice/crio-a0e09c3ac5e722110e418107c9dc47e7564d05895dbc874d458e581db15c6054 WatchSource:0}: Error finding container a0e09c3ac5e722110e418107c9dc47e7564d05895dbc874d458e581db15c6054: Status 404 returned error can't find the container with id a0e09c3ac5e722110e418107c9dc47e7564d05895dbc874d458e581db15c6054
Jul 28 22:17:30.944865 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:30.944833    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-fcfcd7d96-cpvhs" event={"ID":"1b67e040-3901-4cf0-bd2d-3224954ba70e","Type":"ContainerStarted","Data":"aa47faf0e3de2daab27a2a6ee43a672fa839b3c1d1703240b91cec24c584ecf2"}
Jul 28 22:17:30.945897 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:30.945875    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2zd9d" event={"ID":"b164ba31-adb2-42c8-9704-d398ba2650eb","Type":"ContainerStarted","Data":"a0e09c3ac5e722110e418107c9dc47e7564d05895dbc874d458e581db15c6054"}
Jul 28 22:17:32.957730 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:32.957688    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-fcfcd7d96-cpvhs" event={"ID":"1b67e040-3901-4cf0-bd2d-3224954ba70e","Type":"ContainerStarted","Data":"8fd4189cd45dca73a98bb25864b47e3f21cf2b101009745092707664b3612b8a"}
Jul 28 22:17:32.958237 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:32.957814    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-fcfcd7d96-cpvhs"
Jul 28 22:17:32.959142 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:32.959121    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2zd9d" event={"ID":"b164ba31-adb2-42c8-9704-d398ba2650eb","Type":"ContainerStarted","Data":"754ad6825f37429df61c94783a40f767dbf96c5450c45a2b9026d5e29b5a9b37"}
Jul 28 22:17:32.959238 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:32.959167    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2zd9d"
Jul 28 22:17:32.973732 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:32.973687    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-fcfcd7d96-cpvhs" podStartSLOduration=1.887414151 podStartE2EDuration="3.973673399s" totalImagesPullingTime="2.086259248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:17:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:17:32.972290505 +0000 UTC m=+750.352428151" watchObservedRunningTime="2026-07-28 22:17:32.973673399 +0000 UTC m=+750.353811043"
Jul 28 22:17:32.987913 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:32.987876    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2zd9d" podStartSLOduration=1.384048066 podStartE2EDuration="2.98786268s" totalImagesPullingTime="1.603814614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:17:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:17:32.986717222 +0000 UTC m=+750.366854866" watchObservedRunningTime="2026-07-28 22:17:32.98786268 +0000 UTC m=+750.368000325"
Jul 28 22:17:38.968881 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:38.968848    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-fcfcd7d96-cpvhs"
Jul 28 22:17:43.965520 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:43.965488    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2zd9d"
Jul 28 22:17:46.278607 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:46.278569    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq"]
Jul 28 22:17:46.285791 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:46.285747    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq"
Jul 28 22:17:46.289679 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:46.289607    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq"]
Jul 28 22:17:46.289903 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:46.289884    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 28 22:17:46.289989 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:46.289943    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-lntbc\""
Jul 28 22:17:46.290046 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:46.289886    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 28 22:17:46.290174 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:46.290156    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 28 22:17:46.391365 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:46.391321    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txmdk\" (UniqueName: \"kubernetes.io/projected/7f8f63be-a0e6-4b21-912d-839301d1bf0c-kube-api-access-txmdk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq\" (UID: \"7f8f63be-a0e6-4b21-912d-839301d1bf0c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq"
Jul 28 22:17:46.391532 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:46.391381    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f8f63be-a0e6-4b21-912d-839301d1bf0c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq\" (UID: \"7f8f63be-a0e6-4b21-912d-839301d1bf0c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq"
Jul 28 22:17:46.391532 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:46.391444    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f8f63be-a0e6-4b21-912d-839301d1bf0c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq\" (UID: \"7f8f63be-a0e6-4b21-912d-839301d1bf0c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq"
Jul 28 22:17:46.391532 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:46.391487    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f8f63be-a0e6-4b21-912d-839301d1bf0c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq\" (UID: \"7f8f63be-a0e6-4b21-912d-839301d1bf0c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq"
Jul 28 22:17:46.391532 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:46.391515    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f8f63be-a0e6-4b21-912d-839301d1bf0c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq\" (UID: \"7f8f63be-a0e6-4b21-912d-839301d1bf0c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq"
Jul 28 22:17:46.391662 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:46.391600    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f8f63be-a0e6-4b21-912d-839301d1bf0c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq\" (UID: \"7f8f63be-a0e6-4b21-912d-839301d1bf0c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq"
Jul 28 22:17:46.492728 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:46.492693    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f8f63be-a0e6-4b21-912d-839301d1bf0c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq\" (UID: \"7f8f63be-a0e6-4b21-912d-839301d1bf0c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq"
Jul 28 22:17:46.492728 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:46.492730    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f8f63be-a0e6-4b21-912d-839301d1bf0c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq\" (UID: \"7f8f63be-a0e6-4b21-912d-839301d1bf0c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq"
Jul 28 22:17:46.492923 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:46.492848    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f8f63be-a0e6-4b21-912d-839301d1bf0c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq\" (UID: \"7f8f63be-a0e6-4b21-912d-839301d1bf0c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq"
Jul 28 22:17:46.492960 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:46.492941    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txmdk\" (UniqueName: \"kubernetes.io/projected/7f8f63be-a0e6-4b21-912d-839301d1bf0c-kube-api-access-txmdk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq\" (UID: \"7f8f63be-a0e6-4b21-912d-839301d1bf0c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq"
Jul 28 22:17:46.492999 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:46.492971    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f8f63be-a0e6-4b21-912d-839301d1bf0c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq\" (UID: \"7f8f63be-a0e6-4b21-912d-839301d1bf0c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq"
Jul 28 22:17:46.492999 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:46.492991    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f8f63be-a0e6-4b21-912d-839301d1bf0c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq\" (UID: \"7f8f63be-a0e6-4b21-912d-839301d1bf0c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq"
Jul 28 22:17:46.493090 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:46.493071    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f8f63be-a0e6-4b21-912d-839301d1bf0c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq\" (UID: \"7f8f63be-a0e6-4b21-912d-839301d1bf0c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq"
Jul 28 22:17:46.493205 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:46.493184    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f8f63be-a0e6-4b21-912d-839301d1bf0c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq\" (UID: \"7f8f63be-a0e6-4b21-912d-839301d1bf0c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq"
Jul 28 22:17:46.493328 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:46.493305    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f8f63be-a0e6-4b21-912d-839301d1bf0c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq\" (UID: \"7f8f63be-a0e6-4b21-912d-839301d1bf0c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq"
Jul 28 22:17:46.495171 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:46.495144    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f8f63be-a0e6-4b21-912d-839301d1bf0c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq\" (UID: \"7f8f63be-a0e6-4b21-912d-839301d1bf0c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq"
Jul 28 22:17:46.495279 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:46.495237    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f8f63be-a0e6-4b21-912d-839301d1bf0c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq\" (UID: \"7f8f63be-a0e6-4b21-912d-839301d1bf0c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq"
Jul 28 22:17:46.500679 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:46.500657    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txmdk\" (UniqueName: \"kubernetes.io/projected/7f8f63be-a0e6-4b21-912d-839301d1bf0c-kube-api-access-txmdk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq\" (UID: \"7f8f63be-a0e6-4b21-912d-839301d1bf0c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq"
Jul 28 22:17:46.597811 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:46.597722    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq"
Jul 28 22:17:46.732642 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:46.732571    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq"]
Jul 28 22:17:46.735404 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:17:46.735372    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f8f63be_a0e6_4b21_912d_839301d1bf0c.slice/crio-1b85d940da3e2e646f1c59b68135124eac4ed4e07e5b669566e5ddbe8f9ff923 WatchSource:0}: Error finding container 1b85d940da3e2e646f1c59b68135124eac4ed4e07e5b669566e5ddbe8f9ff923: Status 404 returned error can't find the container with id 1b85d940da3e2e646f1c59b68135124eac4ed4e07e5b669566e5ddbe8f9ff923
Jul 28 22:17:47.023037 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:47.022997    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq" event={"ID":"7f8f63be-a0e6-4b21-912d-839301d1bf0c","Type":"ContainerStarted","Data":"1b85d940da3e2e646f1c59b68135124eac4ed4e07e5b669566e5ddbe8f9ff923"}
Jul 28 22:17:48.028249 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:48.028149    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq" event={"ID":"7f8f63be-a0e6-4b21-912d-839301d1bf0c","Type":"ContainerStarted","Data":"d73f41cdf8d5cfe50f031ba3a406aa0336b7e7bb2cf7272b55243ab7a1510887"}
Jul 28 22:17:48.046723 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:48.046671    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq" podStartSLOduration=1.011582959 podStartE2EDuration="2.046651512s" totalImagesPullingTime="1.035068553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:17:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:17:48.046119751 +0000 UTC m=+765.426257409" watchObservedRunningTime="2026-07-28 22:17:48.046651512 +0000 UTC m=+765.426789161"
Jul 28 22:17:56.598622 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:56.598583    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq"
Jul 28 22:17:56.599096 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:56.598677    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq"
Jul 28 22:17:56.611417 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:56.611395    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq"
Jul 28 22:17:57.082084 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:17:57.082057    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq"
Jul 28 22:18:08.948179 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:08.948144    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-lppq7"]
Jul 28 22:18:08.954246 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:08.954219    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lppq7"
Jul 28 22:18:08.957073 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:08.957047    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 28 22:18:08.961938 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:08.961913    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-lppq7"]
Jul 28 22:18:09.084427 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:09.084390    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acd46159-9704-48f8-b0ec-a5855c2c804a-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lppq7\" (UID: \"acd46159-9704-48f8-b0ec-a5855c2c804a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lppq7"
Jul 28 22:18:09.084593 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:09.084473    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acd46159-9704-48f8-b0ec-a5855c2c804a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lppq7\" (UID: \"acd46159-9704-48f8-b0ec-a5855c2c804a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lppq7"
Jul 28 22:18:09.084593 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:09.084514    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl84l\" (UniqueName: \"kubernetes.io/projected/acd46159-9704-48f8-b0ec-a5855c2c804a-kube-api-access-cl84l\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lppq7\" (UID: \"acd46159-9704-48f8-b0ec-a5855c2c804a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lppq7"
Jul 28 22:18:09.084593 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:09.084546    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acd46159-9704-48f8-b0ec-a5855c2c804a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lppq7\" (UID: \"acd46159-9704-48f8-b0ec-a5855c2c804a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lppq7"
Jul 28 22:18:09.084593 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:09.084565    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acd46159-9704-48f8-b0ec-a5855c2c804a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lppq7\" (UID: \"acd46159-9704-48f8-b0ec-a5855c2c804a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lppq7"
Jul 28 22:18:09.084735 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:09.084663    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acd46159-9704-48f8-b0ec-a5855c2c804a-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lppq7\" (UID: \"acd46159-9704-48f8-b0ec-a5855c2c804a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lppq7"
Jul 28 22:18:09.185852 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:09.185826    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acd46159-9704-48f8-b0ec-a5855c2c804a-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lppq7\" (UID: \"acd46159-9704-48f8-b0ec-a5855c2c804a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lppq7"
Jul 28 22:18:09.186014 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:09.185858    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acd46159-9704-48f8-b0ec-a5855c2c804a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lppq7\" (UID: \"acd46159-9704-48f8-b0ec-a5855c2c804a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lppq7"
Jul 28 22:18:09.186014 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:09.185876    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl84l\" (UniqueName: \"kubernetes.io/projected/acd46159-9704-48f8-b0ec-a5855c2c804a-kube-api-access-cl84l\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lppq7\" (UID: \"acd46159-9704-48f8-b0ec-a5855c2c804a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lppq7"
Jul 28 22:18:09.186014 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:09.185902    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acd46159-9704-48f8-b0ec-a5855c2c804a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lppq7\" (UID: \"acd46159-9704-48f8-b0ec-a5855c2c804a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lppq7"
Jul 28 22:18:09.186014 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:09.185928    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acd46159-9704-48f8-b0ec-a5855c2c804a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lppq7\" (UID: \"acd46159-9704-48f8-b0ec-a5855c2c804a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lppq7"
Jul 28 22:18:09.186014 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:09.185988    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acd46159-9704-48f8-b0ec-a5855c2c804a-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lppq7\" (UID: \"acd46159-9704-48f8-b0ec-a5855c2c804a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lppq7"
Jul 28 22:18:09.186286 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:09.186267    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acd46159-9704-48f8-b0ec-a5855c2c804a-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lppq7\" (UID: \"acd46159-9704-48f8-b0ec-a5855c2c804a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lppq7"
Jul 28 22:18:09.186363 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:09.186283    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acd46159-9704-48f8-b0ec-a5855c2c804a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lppq7\" (UID: \"acd46159-9704-48f8-b0ec-a5855c2c804a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lppq7"
Jul 28 22:18:09.186444 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:09.186422    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acd46159-9704-48f8-b0ec-a5855c2c804a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lppq7\" (UID: \"acd46159-9704-48f8-b0ec-a5855c2c804a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lppq7"
Jul 28 22:18:09.188196 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:09.188170    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acd46159-9704-48f8-b0ec-a5855c2c804a-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lppq7\" (UID: \"acd46159-9704-48f8-b0ec-a5855c2c804a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lppq7"
Jul 28 22:18:09.188603 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:09.188582    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acd46159-9704-48f8-b0ec-a5855c2c804a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lppq7\" (UID: \"acd46159-9704-48f8-b0ec-a5855c2c804a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lppq7"
Jul 28 22:18:09.192936 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:09.192917    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl84l\" (UniqueName: \"kubernetes.io/projected/acd46159-9704-48f8-b0ec-a5855c2c804a-kube-api-access-cl84l\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lppq7\" (UID: \"acd46159-9704-48f8-b0ec-a5855c2c804a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lppq7"
Jul 28 22:18:09.265835 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:09.265759    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lppq7"
Jul 28 22:18:09.399579 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:09.399548    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-lppq7"]
Jul 28 22:18:09.403000 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:18:09.402969    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacd46159_9704_48f8_b0ec_a5855c2c804a.slice/crio-6d27d4c11b8b05321f2df4fdc571c2237a479ed8d4084e01d1a95a7d264b0938 WatchSource:0}: Error finding container 6d27d4c11b8b05321f2df4fdc571c2237a479ed8d4084e01d1a95a7d264b0938: Status 404 returned error can't find the container with id 6d27d4c11b8b05321f2df4fdc571c2237a479ed8d4084e01d1a95a7d264b0938
Jul 28 22:18:10.128437 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:10.128405    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lppq7" event={"ID":"acd46159-9704-48f8-b0ec-a5855c2c804a","Type":"ContainerStarted","Data":"d0bd32706e0d14bb4603a82f02df03618d4b6f2f1d43a0b522a9a26dd15ab107"}
Jul 28 22:18:10.128891 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:10.128448    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lppq7" event={"ID":"acd46159-9704-48f8-b0ec-a5855c2c804a","Type":"ContainerStarted","Data":"6d27d4c11b8b05321f2df4fdc571c2237a479ed8d4084e01d1a95a7d264b0938"}
Jul 28 22:18:10.150489 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:10.150430    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lppq7" podStartSLOduration=1.958787026 podStartE2EDuration="2.150413951s" totalImagesPullingTime="191.626925ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:18:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:18:10.148732999 +0000 UTC m=+787.528870643" watchObservedRunningTime="2026-07-28 22:18:10.150413951 +0000 UTC m=+787.530551595"
Jul 28 22:18:19.265964 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:19.265929    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lppq7"
Jul 28 22:18:19.266378 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:19.265993    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lppq7"
Jul 28 22:18:19.278639 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:19.278611    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lppq7"
Jul 28 22:18:20.188597 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:20.188571    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lppq7"
Jul 28 22:18:37.765559 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:37.765528    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-svj7p"]
Jul 28 22:18:37.769333 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:37.769300    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-svj7p"
Jul 28 22:18:37.772014 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:37.771990    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 28 22:18:37.780561 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:37.780537    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-svj7p"]
Jul 28 22:18:37.942134 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:37.942102    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/862c1e2f-c706-4da3-a357-530decd9458a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-svj7p\" (UID: \"862c1e2f-c706-4da3-a357-530decd9458a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-svj7p"
Jul 28 22:18:37.942134 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:37.942135    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/862c1e2f-c706-4da3-a357-530decd9458a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-svj7p\" (UID: \"862c1e2f-c706-4da3-a357-530decd9458a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-svj7p"
Jul 28 22:18:37.942378 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:37.942154    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/862c1e2f-c706-4da3-a357-530decd9458a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-svj7p\" (UID: \"862c1e2f-c706-4da3-a357-530decd9458a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-svj7p"
Jul 28 22:18:37.942378 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:37.942259    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/862c1e2f-c706-4da3-a357-530decd9458a-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-svj7p\" (UID: \"862c1e2f-c706-4da3-a357-530decd9458a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-svj7p"
Jul 28 22:18:37.942378 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:37.942295    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/862c1e2f-c706-4da3-a357-530decd9458a-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-svj7p\" (UID: \"862c1e2f-c706-4da3-a357-530decd9458a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-svj7p"
Jul 28 22:18:37.942378 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:37.942313    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74kdq\" (UniqueName: \"kubernetes.io/projected/862c1e2f-c706-4da3-a357-530decd9458a-kube-api-access-74kdq\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-svj7p\" (UID: \"862c1e2f-c706-4da3-a357-530decd9458a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-svj7p"
Jul 28 22:18:38.043315 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:38.043236    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/862c1e2f-c706-4da3-a357-530decd9458a-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-svj7p\" (UID: \"862c1e2f-c706-4da3-a357-530decd9458a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-svj7p"
Jul 28 22:18:38.043315 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:38.043280    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/862c1e2f-c706-4da3-a357-530decd9458a-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-svj7p\" (UID: \"862c1e2f-c706-4da3-a357-530decd9458a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-svj7p"
Jul 28 22:18:38.043315 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:38.043298    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74kdq\" (UniqueName: \"kubernetes.io/projected/862c1e2f-c706-4da3-a357-530decd9458a-kube-api-access-74kdq\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-svj7p\" (UID: \"862c1e2f-c706-4da3-a357-530decd9458a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-svj7p"
Jul 28 22:18:38.043625 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:38.043386    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/862c1e2f-c706-4da3-a357-530decd9458a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-svj7p\" (UID: \"862c1e2f-c706-4da3-a357-530decd9458a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-svj7p"
Jul 28 22:18:38.043625 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:38.043413    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/862c1e2f-c706-4da3-a357-530decd9458a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-svj7p\" (UID: \"862c1e2f-c706-4da3-a357-530decd9458a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-svj7p"
Jul 28 22:18:38.043625 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:38.043437    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/862c1e2f-c706-4da3-a357-530decd9458a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-svj7p\" (UID: \"862c1e2f-c706-4da3-a357-530decd9458a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-svj7p"
Jul 28 22:18:38.043787 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:38.043710    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/862c1e2f-c706-4da3-a357-530decd9458a-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-svj7p\" (UID: \"862c1e2f-c706-4da3-a357-530decd9458a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-svj7p"
Jul 28 22:18:38.043841 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:38.043817    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/862c1e2f-c706-4da3-a357-530decd9458a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-svj7p\" (UID: \"862c1e2f-c706-4da3-a357-530decd9458a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-svj7p"
Jul 28 22:18:38.043841 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:38.043834    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/862c1e2f-c706-4da3-a357-530decd9458a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-svj7p\" (UID: \"862c1e2f-c706-4da3-a357-530decd9458a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-svj7p"
Jul 28 22:18:38.045655 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:38.045630    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/862c1e2f-c706-4da3-a357-530decd9458a-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-svj7p\" (UID: \"862c1e2f-c706-4da3-a357-530decd9458a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-svj7p"
Jul 28 22:18:38.045845 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:38.045829    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/862c1e2f-c706-4da3-a357-530decd9458a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-svj7p\" (UID: \"862c1e2f-c706-4da3-a357-530decd9458a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-svj7p"
Jul 28 22:18:38.051151 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:38.051128    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74kdq\" (UniqueName: \"kubernetes.io/projected/862c1e2f-c706-4da3-a357-530decd9458a-kube-api-access-74kdq\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-svj7p\" (UID: \"862c1e2f-c706-4da3-a357-530decd9458a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-svj7p"
Jul 28 22:18:38.079988 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:38.079962    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-svj7p"
Jul 28 22:18:38.213582 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:38.213439    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-svj7p"]
Jul 28 22:18:38.215921 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:18:38.215889    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod862c1e2f_c706_4da3_a357_530decd9458a.slice/crio-9ed569fb2f4ac84a1a5e95986145d05e80969c9da4c48d9a8df620e817af87dd WatchSource:0}: Error finding container 9ed569fb2f4ac84a1a5e95986145d05e80969c9da4c48d9a8df620e817af87dd: Status 404 returned error can't find the container with id 9ed569fb2f4ac84a1a5e95986145d05e80969c9da4c48d9a8df620e817af87dd
Jul 28 22:18:38.263296 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:38.263263    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-svj7p" event={"ID":"862c1e2f-c706-4da3-a357-530decd9458a","Type":"ContainerStarted","Data":"9ed569fb2f4ac84a1a5e95986145d05e80969c9da4c48d9a8df620e817af87dd"}
Jul 28 22:18:39.268983 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:39.268943    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-svj7p" event={"ID":"862c1e2f-c706-4da3-a357-530decd9458a","Type":"ContainerStarted","Data":"9a4331066b6fe912bb41feb6a99977080b7acd31e7cf25d8d43505234185bc4b"}
Jul 28 22:18:39.286759 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:39.286710    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-svj7p" podStartSLOduration=2.11789372 podStartE2EDuration="2.286693032s" totalImagesPullingTime="168.799312ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:18:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:18:39.284976812 +0000 UTC m=+816.665114456" watchObservedRunningTime="2026-07-28 22:18:39.286693032 +0000 UTC m=+816.666830676"
Jul 28 22:18:48.080555 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:48.080518    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-svj7p"
Jul 28 22:18:48.080984 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:48.080581    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-svj7p"
Jul 28 22:18:48.092961 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:48.092929    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-svj7p"
Jul 28 22:18:48.326011 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:18:48.325983    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-svj7p"
Jul 28 22:20:03.039536 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:20:03.039458    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mcj8p_7bab32b0-13c6-4055-916e-a7454bafb5b6/ovn-acl-logging/0.log"
Jul 28 22:20:03.040099 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:20:03.039972    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mcj8p_7bab32b0-13c6-4055-916e-a7454bafb5b6/ovn-acl-logging/0.log"
Jul 28 22:20:44.949788 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:20:44.949747    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8687496d99-pzsps"]
Jul 28 22:20:44.950332 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:20:44.950019    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-8687496d99-pzsps" podUID="856899f8-c889-4c7b-acca-7b773b624420" containerName="manager" containerID="cri-o://38ea543c5da771e80d648ed25edf6d441c2280bc190d00d6793c3d93d80e2da6" gracePeriod=30
Jul 28 22:20:44.998962 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:20:44.998930    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod856899f8_c889_4c7b_acca_7b773b624420.slice/crio-conmon-38ea543c5da771e80d648ed25edf6d441c2280bc190d00d6793c3d93d80e2da6.scope\": RecentStats: unable to find data in memory cache]"
Jul 28 22:20:45.200824 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:20:45.200762    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8687496d99-pzsps"
Jul 28 22:20:45.256492 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:20:45.256462    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/856899f8-c889-4c7b-acca-7b773b624420-webhook-cert\") pod \"856899f8-c889-4c7b-acca-7b773b624420\" (UID: \"856899f8-c889-4c7b-acca-7b773b624420\") "
Jul 28 22:20:45.256666 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:20:45.256513    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7sjvk\" (UniqueName: \"kubernetes.io/projected/856899f8-c889-4c7b-acca-7b773b624420-kube-api-access-7sjvk\") pod \"856899f8-c889-4c7b-acca-7b773b624420\" (UID: \"856899f8-c889-4c7b-acca-7b773b624420\") "
Jul 28 22:20:45.258593 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:20:45.258559    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/856899f8-c889-4c7b-acca-7b773b624420-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "856899f8-c889-4c7b-acca-7b773b624420" (UID: "856899f8-c889-4c7b-acca-7b773b624420"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:20:45.258713 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:20:45.258646    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/856899f8-c889-4c7b-acca-7b773b624420-kube-api-access-7sjvk" (OuterVolumeSpecName: "kube-api-access-7sjvk") pod "856899f8-c889-4c7b-acca-7b773b624420" (UID: "856899f8-c889-4c7b-acca-7b773b624420"). InnerVolumeSpecName "kube-api-access-7sjvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:20:45.357979 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:20:45.357933    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/856899f8-c889-4c7b-acca-7b773b624420-webhook-cert\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:20:45.357979 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:20:45.357975    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sjvk\" (UniqueName: \"kubernetes.io/projected/856899f8-c889-4c7b-acca-7b773b624420-kube-api-access-7sjvk\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:20:45.856458 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:20:45.856423    2563 generic.go:356] "Generic (PLEG): container finished" podID="856899f8-c889-4c7b-acca-7b773b624420" containerID="38ea543c5da771e80d648ed25edf6d441c2280bc190d00d6793c3d93d80e2da6" exitCode=0
Jul 28 22:20:45.856641 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:20:45.856476    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8687496d99-pzsps"
Jul 28 22:20:45.856641 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:20:45.856515    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8687496d99-pzsps" event={"ID":"856899f8-c889-4c7b-acca-7b773b624420","Type":"ContainerDied","Data":"38ea543c5da771e80d648ed25edf6d441c2280bc190d00d6793c3d93d80e2da6"}
Jul 28 22:20:45.856641 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:20:45.856569    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8687496d99-pzsps" event={"ID":"856899f8-c889-4c7b-acca-7b773b624420","Type":"ContainerDied","Data":"0f854fa0aec70a5d99c8700c7e9f939a0b044fdcb3727bd5a42e4fd51d120cbe"}
Jul 28 22:20:45.856641 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:20:45.856594    2563 scope.go:117] "RemoveContainer" containerID="38ea543c5da771e80d648ed25edf6d441c2280bc190d00d6793c3d93d80e2da6"
Jul 28 22:20:45.865392 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:20:45.865373    2563 scope.go:117] "RemoveContainer" containerID="38ea543c5da771e80d648ed25edf6d441c2280bc190d00d6793c3d93d80e2da6"
Jul 28 22:20:45.865633 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:20:45.865615    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38ea543c5da771e80d648ed25edf6d441c2280bc190d00d6793c3d93d80e2da6\": container with ID starting with 38ea543c5da771e80d648ed25edf6d441c2280bc190d00d6793c3d93d80e2da6 not found: ID does not exist" containerID="38ea543c5da771e80d648ed25edf6d441c2280bc190d00d6793c3d93d80e2da6"
Jul 28 22:20:45.865689 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:20:45.865641    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38ea543c5da771e80d648ed25edf6d441c2280bc190d00d6793c3d93d80e2da6"} err="failed to get container status \"38ea543c5da771e80d648ed25edf6d441c2280bc190d00d6793c3d93d80e2da6\": rpc error: code = NotFound desc = could not find container \"38ea543c5da771e80d648ed25edf6d441c2280bc190d00d6793c3d93d80e2da6\": container with ID starting with 38ea543c5da771e80d648ed25edf6d441c2280bc190d00d6793c3d93d80e2da6 not found: ID does not exist"
Jul 28 22:20:45.878671 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:20:45.878639    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8687496d99-pzsps"]
Jul 28 22:20:45.883068 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:20:45.883048    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-8687496d99-pzsps"]
Jul 28 22:20:47.134382 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:20:47.131282    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="856899f8-c889-4c7b-acca-7b773b624420" path="/var/lib/kubelet/pods/856899f8-c889-4c7b-acca-7b773b624420/volumes"
Jul 28 22:21:46.696009 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:21:46.695972    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8687496d99-xswxv"]
Jul 28 22:21:46.696459 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:21:46.696370    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="856899f8-c889-4c7b-acca-7b773b624420" containerName="manager"
Jul 28 22:21:46.696459 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:21:46.696382    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="856899f8-c889-4c7b-acca-7b773b624420" containerName="manager"
Jul 28 22:21:46.696459 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:21:46.696442    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="856899f8-c889-4c7b-acca-7b773b624420" containerName="manager"
Jul 28 22:21:46.699456 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:21:46.699439    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8687496d99-xswxv"
Jul 28 22:21:46.702809 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:21:46.702786    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 28 22:21:46.702967 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:21:46.702908    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mtsxn\""
Jul 28 22:21:46.702967 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:21:46.702931    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 28 22:21:46.707307 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:21:46.707282    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8687496d99-xswxv"]
Jul 28 22:21:46.707473 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:21:46.707446    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gdt8\" (UniqueName: \"kubernetes.io/projected/e68e3d37-fbfb-48b9-bc2c-cd2a802b0331-kube-api-access-7gdt8\") pod \"maas-controller-8687496d99-xswxv\" (UID: \"e68e3d37-fbfb-48b9-bc2c-cd2a802b0331\") " pod="opendatahub/maas-controller-8687496d99-xswxv"
Jul 28 22:21:46.707588 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:21:46.707506    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e68e3d37-fbfb-48b9-bc2c-cd2a802b0331-webhook-cert\") pod \"maas-controller-8687496d99-xswxv\" (UID: \"e68e3d37-fbfb-48b9-bc2c-cd2a802b0331\") " pod="opendatahub/maas-controller-8687496d99-xswxv"
Jul 28 22:21:46.808573 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:21:46.808544    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gdt8\" (UniqueName: \"kubernetes.io/projected/e68e3d37-fbfb-48b9-bc2c-cd2a802b0331-kube-api-access-7gdt8\") pod \"maas-controller-8687496d99-xswxv\" (UID: \"e68e3d37-fbfb-48b9-bc2c-cd2a802b0331\") " pod="opendatahub/maas-controller-8687496d99-xswxv"
Jul 28 22:21:46.808573 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:21:46.808583    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e68e3d37-fbfb-48b9-bc2c-cd2a802b0331-webhook-cert\") pod \"maas-controller-8687496d99-xswxv\" (UID: \"e68e3d37-fbfb-48b9-bc2c-cd2a802b0331\") " pod="opendatahub/maas-controller-8687496d99-xswxv"
Jul 28 22:21:46.810987 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:21:46.810964    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e68e3d37-fbfb-48b9-bc2c-cd2a802b0331-webhook-cert\") pod \"maas-controller-8687496d99-xswxv\" (UID: \"e68e3d37-fbfb-48b9-bc2c-cd2a802b0331\") " pod="opendatahub/maas-controller-8687496d99-xswxv"
Jul 28 22:21:46.817498 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:21:46.817478    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gdt8\" (UniqueName: \"kubernetes.io/projected/e68e3d37-fbfb-48b9-bc2c-cd2a802b0331-kube-api-access-7gdt8\") pod \"maas-controller-8687496d99-xswxv\" (UID: \"e68e3d37-fbfb-48b9-bc2c-cd2a802b0331\") " pod="opendatahub/maas-controller-8687496d99-xswxv"
Jul 28 22:21:47.010847 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:21:47.010756    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8687496d99-xswxv"
Jul 28 22:21:47.146666 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:21:47.146638    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8687496d99-xswxv"]
Jul 28 22:21:47.149738 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:21:47.149708    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode68e3d37_fbfb_48b9_bc2c_cd2a802b0331.slice/crio-2cee8bedb51bd66b6dd777ce045cc8e14e314585ef847ce1c53f9aa342456820 WatchSource:0}: Error finding container 2cee8bedb51bd66b6dd777ce045cc8e14e314585ef847ce1c53f9aa342456820: Status 404 returned error can't find the container with id 2cee8bedb51bd66b6dd777ce045cc8e14e314585ef847ce1c53f9aa342456820
Jul 28 22:21:47.151393 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:21:47.151372    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 22:21:48.153010 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:21:48.152968    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8687496d99-xswxv" event={"ID":"e68e3d37-fbfb-48b9-bc2c-cd2a802b0331","Type":"ContainerStarted","Data":"0c74aecfaf87f966fee463ca812955e1e8eee18252c6a97ed044d3427636abab"}
Jul 28 22:21:48.153440 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:21:48.153035    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8687496d99-xswxv" event={"ID":"e68e3d37-fbfb-48b9-bc2c-cd2a802b0331","Type":"ContainerStarted","Data":"2cee8bedb51bd66b6dd777ce045cc8e14e314585ef847ce1c53f9aa342456820"}
Jul 28 22:21:48.153440 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:21:48.153213    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8687496d99-xswxv"
Jul 28 22:21:48.172376 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:21:48.172305    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-8687496d99-xswxv" podStartSLOduration=1.774110268 podStartE2EDuration="2.172288581s" totalImagesPullingTime="398.178313ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:21:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:21:48.170253991 +0000 UTC m=+1005.550391635" watchObservedRunningTime="2026-07-28 22:21:48.172288581 +0000 UTC m=+1005.552426251"
Jul 28 22:21:59.159457 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:21:59.159422    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8687496d99-xswxv"
Jul 28 22:22:03.083989 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:03.083954    2563 scope.go:117] "RemoveContainer" containerID="d3b2fb084816435569fbef006ab220cff7ea7e673502bd3c0e9233f6e1fc4337"
Jul 28 22:22:03.805224 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:03.805186    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8687496d99-xswxv"]
Jul 28 22:22:03.805448 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:03.805409    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-8687496d99-xswxv" podUID="e68e3d37-fbfb-48b9-bc2c-cd2a802b0331" containerName="manager" containerID="cri-o://0c74aecfaf87f966fee463ca812955e1e8eee18252c6a97ed044d3427636abab" gracePeriod=30
Jul 28 22:22:04.055901 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:04.055843    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8687496d99-xswxv"
Jul 28 22:22:04.160984 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:04.160952    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e68e3d37-fbfb-48b9-bc2c-cd2a802b0331-webhook-cert\") pod \"e68e3d37-fbfb-48b9-bc2c-cd2a802b0331\" (UID: \"e68e3d37-fbfb-48b9-bc2c-cd2a802b0331\") "
Jul 28 22:22:04.161466 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:04.161017    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gdt8\" (UniqueName: \"kubernetes.io/projected/e68e3d37-fbfb-48b9-bc2c-cd2a802b0331-kube-api-access-7gdt8\") pod \"e68e3d37-fbfb-48b9-bc2c-cd2a802b0331\" (UID: \"e68e3d37-fbfb-48b9-bc2c-cd2a802b0331\") "
Jul 28 22:22:04.163046 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:04.163021    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e68e3d37-fbfb-48b9-bc2c-cd2a802b0331-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e68e3d37-fbfb-48b9-bc2c-cd2a802b0331" (UID: "e68e3d37-fbfb-48b9-bc2c-cd2a802b0331"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:22:04.163182 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:04.163163    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e68e3d37-fbfb-48b9-bc2c-cd2a802b0331-kube-api-access-7gdt8" (OuterVolumeSpecName: "kube-api-access-7gdt8") pod "e68e3d37-fbfb-48b9-bc2c-cd2a802b0331" (UID: "e68e3d37-fbfb-48b9-bc2c-cd2a802b0331"). InnerVolumeSpecName "kube-api-access-7gdt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:22:04.230194 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:04.230166    2563 generic.go:356] "Generic (PLEG): container finished" podID="e68e3d37-fbfb-48b9-bc2c-cd2a802b0331" containerID="0c74aecfaf87f966fee463ca812955e1e8eee18252c6a97ed044d3427636abab" exitCode=0
Jul 28 22:22:04.230331 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:04.230209    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8687496d99-xswxv"
Jul 28 22:22:04.230331 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:04.230246    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8687496d99-xswxv" event={"ID":"e68e3d37-fbfb-48b9-bc2c-cd2a802b0331","Type":"ContainerDied","Data":"0c74aecfaf87f966fee463ca812955e1e8eee18252c6a97ed044d3427636abab"}
Jul 28 22:22:04.230331 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:04.230288    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8687496d99-xswxv" event={"ID":"e68e3d37-fbfb-48b9-bc2c-cd2a802b0331","Type":"ContainerDied","Data":"2cee8bedb51bd66b6dd777ce045cc8e14e314585ef847ce1c53f9aa342456820"}
Jul 28 22:22:04.230331 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:04.230309    2563 scope.go:117] "RemoveContainer" containerID="0c74aecfaf87f966fee463ca812955e1e8eee18252c6a97ed044d3427636abab"
Jul 28 22:22:04.239551 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:04.239534    2563 scope.go:117] "RemoveContainer" containerID="0c74aecfaf87f966fee463ca812955e1e8eee18252c6a97ed044d3427636abab"
Jul 28 22:22:04.239871 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:22:04.239848    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c74aecfaf87f966fee463ca812955e1e8eee18252c6a97ed044d3427636abab\": container with ID starting with 0c74aecfaf87f966fee463ca812955e1e8eee18252c6a97ed044d3427636abab not found: ID does not exist" containerID="0c74aecfaf87f966fee463ca812955e1e8eee18252c6a97ed044d3427636abab"
Jul 28 22:22:04.239950 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:04.239878    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c74aecfaf87f966fee463ca812955e1e8eee18252c6a97ed044d3427636abab"} err="failed to get container status \"0c74aecfaf87f966fee463ca812955e1e8eee18252c6a97ed044d3427636abab\": rpc error: code = NotFound desc = could not find container \"0c74aecfaf87f966fee463ca812955e1e8eee18252c6a97ed044d3427636abab\": container with ID starting with 0c74aecfaf87f966fee463ca812955e1e8eee18252c6a97ed044d3427636abab not found: ID does not exist"
Jul 28 22:22:04.252734 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:04.252709    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8687496d99-xswxv"]
Jul 28 22:22:04.256262 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:04.256239    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-8687496d99-xswxv"]
Jul 28 22:22:04.261768 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:04.261747    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gdt8\" (UniqueName: \"kubernetes.io/projected/e68e3d37-fbfb-48b9-bc2c-cd2a802b0331-kube-api-access-7gdt8\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:22:04.261824 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:04.261772    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e68e3d37-fbfb-48b9-bc2c-cd2a802b0331-webhook-cert\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:22:05.057988 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:05.057956    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8687496d99-crms2"]
Jul 28 22:22:05.058301 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:05.058289    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e68e3d37-fbfb-48b9-bc2c-cd2a802b0331" containerName="manager"
Jul 28 22:22:05.058347 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:05.058302    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e68e3d37-fbfb-48b9-bc2c-cd2a802b0331" containerName="manager"
Jul 28 22:22:05.058402 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:05.058374    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e68e3d37-fbfb-48b9-bc2c-cd2a802b0331" containerName="manager"
Jul 28 22:22:05.062454 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:05.062437    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8687496d99-crms2"
Jul 28 22:22:05.064748 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:05.064727    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 28 22:22:05.065746 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:05.065727    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mtsxn\""
Jul 28 22:22:05.065849 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:05.065761    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 28 22:22:05.069304 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:05.069280    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8687496d99-crms2"]
Jul 28 22:22:05.130279 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:05.130249    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e68e3d37-fbfb-48b9-bc2c-cd2a802b0331" path="/var/lib/kubelet/pods/e68e3d37-fbfb-48b9-bc2c-cd2a802b0331/volumes"
Jul 28 22:22:05.169790 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:05.169757    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87s8d\" (UniqueName: \"kubernetes.io/projected/07403f33-2401-479b-8b78-3b0c0b2716e9-kube-api-access-87s8d\") pod \"maas-controller-8687496d99-crms2\" (UID: \"07403f33-2401-479b-8b78-3b0c0b2716e9\") " pod="opendatahub/maas-controller-8687496d99-crms2"
Jul 28 22:22:05.170152 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:05.169859    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07403f33-2401-479b-8b78-3b0c0b2716e9-webhook-cert\") pod \"maas-controller-8687496d99-crms2\" (UID: \"07403f33-2401-479b-8b78-3b0c0b2716e9\") " pod="opendatahub/maas-controller-8687496d99-crms2"
Jul 28 22:22:05.270543 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:05.270494    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07403f33-2401-479b-8b78-3b0c0b2716e9-webhook-cert\") pod \"maas-controller-8687496d99-crms2\" (UID: \"07403f33-2401-479b-8b78-3b0c0b2716e9\") " pod="opendatahub/maas-controller-8687496d99-crms2"
Jul 28 22:22:05.270736 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:05.270676    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87s8d\" (UniqueName: \"kubernetes.io/projected/07403f33-2401-479b-8b78-3b0c0b2716e9-kube-api-access-87s8d\") pod \"maas-controller-8687496d99-crms2\" (UID: \"07403f33-2401-479b-8b78-3b0c0b2716e9\") " pod="opendatahub/maas-controller-8687496d99-crms2"
Jul 28 22:22:05.272943 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:05.272914    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07403f33-2401-479b-8b78-3b0c0b2716e9-webhook-cert\") pod \"maas-controller-8687496d99-crms2\" (UID: \"07403f33-2401-479b-8b78-3b0c0b2716e9\") " pod="opendatahub/maas-controller-8687496d99-crms2"
Jul 28 22:22:05.278263 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:05.278238    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87s8d\" (UniqueName: \"kubernetes.io/projected/07403f33-2401-479b-8b78-3b0c0b2716e9-kube-api-access-87s8d\") pod \"maas-controller-8687496d99-crms2\" (UID: \"07403f33-2401-479b-8b78-3b0c0b2716e9\") " pod="opendatahub/maas-controller-8687496d99-crms2"
Jul 28 22:22:05.374386 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:05.374293    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8687496d99-crms2"
Jul 28 22:22:05.505079 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:05.504937    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8687496d99-crms2"]
Jul 28 22:22:05.507600 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:22:05.507574    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07403f33_2401_479b_8b78_3b0c0b2716e9.slice/crio-0899b9a1fa54084069e64ea84f0219ac94cafea118b9ee0f8dd901e1a4138cea WatchSource:0}: Error finding container 0899b9a1fa54084069e64ea84f0219ac94cafea118b9ee0f8dd901e1a4138cea: Status 404 returned error can't find the container with id 0899b9a1fa54084069e64ea84f0219ac94cafea118b9ee0f8dd901e1a4138cea
Jul 28 22:22:06.247987 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:06.247892    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8687496d99-crms2" event={"ID":"07403f33-2401-479b-8b78-3b0c0b2716e9","Type":"ContainerStarted","Data":"6124686310ca1d4e21d96a1bf3e1bd7ff9bbb1690a9a3e10b3ea64921da92de0"}
Jul 28 22:22:06.247987 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:06.247946    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8687496d99-crms2" event={"ID":"07403f33-2401-479b-8b78-3b0c0b2716e9","Type":"ContainerStarted","Data":"0899b9a1fa54084069e64ea84f0219ac94cafea118b9ee0f8dd901e1a4138cea"}
Jul 28 22:22:06.248420 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:06.247983    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8687496d99-crms2"
Jul 28 22:22:06.266384 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:06.266318    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-8687496d99-crms2" podStartSLOduration=0.839719631 podStartE2EDuration="1.266302178s" totalImagesPullingTime="426.582547ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:22:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:22:06.26434702 +0000 UTC m=+1023.644484668" watchObservedRunningTime="2026-07-28 22:22:06.266302178 +0000 UTC m=+1023.646439822"
Jul 28 22:22:17.254619 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:22:17.254591    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8687496d99-crms2"
Jul 28 22:25:03.066457 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:25:03.066425    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mcj8p_7bab32b0-13c6-4055-916e-a7454bafb5b6/ovn-acl-logging/0.log"
Jul 28 22:25:03.067380 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:25:03.067336    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mcj8p_7bab32b0-13c6-4055-916e-a7454bafb5b6/ovn-acl-logging/0.log"
Jul 28 22:27:08.871072 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:27:08.871037    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q6s9l"]
Jul 28 22:27:08.871784 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:27:08.871237    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q6s9l" podUID="ad2a4a21-a0b5-4b90-ae3d-c29e89c26d01" containerName="manager" containerID="cri-o://94a5462239650c437ad5fe18f3cc20efdacb9436edefad8e07bb169030716d48" gracePeriod=10
Jul 28 22:27:11.618178 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:27:11.618142    2563 generic.go:356] "Generic (PLEG): container finished" podID="ad2a4a21-a0b5-4b90-ae3d-c29e89c26d01" containerID="94a5462239650c437ad5fe18f3cc20efdacb9436edefad8e07bb169030716d48" exitCode=0
Jul 28 22:27:11.618535 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:27:11.618216    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q6s9l" event={"ID":"ad2a4a21-a0b5-4b90-ae3d-c29e89c26d01","Type":"ContainerDied","Data":"94a5462239650c437ad5fe18f3cc20efdacb9436edefad8e07bb169030716d48"}
Jul 28 22:27:11.618535 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:27:11.618267    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q6s9l" event={"ID":"ad2a4a21-a0b5-4b90-ae3d-c29e89c26d01","Type":"ContainerDied","Data":"8d6ceb86bb76c383d0a1dfb80189c8175ae5492f81235d8b92fca772631e3b22"}
Jul 28 22:27:11.618535 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:27:11.618291    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d6ceb86bb76c383d0a1dfb80189c8175ae5492f81235d8b92fca772631e3b22"
Jul 28 22:27:11.620993 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:27:11.620972    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q6s9l"
Jul 28 22:27:11.732347 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:27:11.732316    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad2a4a21-a0b5-4b90-ae3d-c29e89c26d01-extensions-socket-volume\") pod \"ad2a4a21-a0b5-4b90-ae3d-c29e89c26d01\" (UID: \"ad2a4a21-a0b5-4b90-ae3d-c29e89c26d01\") "
Jul 28 22:27:11.732544 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:27:11.732401    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hg8d\" (UniqueName: \"kubernetes.io/projected/ad2a4a21-a0b5-4b90-ae3d-c29e89c26d01-kube-api-access-9hg8d\") pod \"ad2a4a21-a0b5-4b90-ae3d-c29e89c26d01\" (UID: \"ad2a4a21-a0b5-4b90-ae3d-c29e89c26d01\") "
Jul 28 22:27:11.732688 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:27:11.732661    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad2a4a21-a0b5-4b90-ae3d-c29e89c26d01-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ad2a4a21-a0b5-4b90-ae3d-c29e89c26d01" (UID: "ad2a4a21-a0b5-4b90-ae3d-c29e89c26d01"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:27:11.734414 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:27:11.734363    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad2a4a21-a0b5-4b90-ae3d-c29e89c26d01-kube-api-access-9hg8d" (OuterVolumeSpecName: "kube-api-access-9hg8d") pod "ad2a4a21-a0b5-4b90-ae3d-c29e89c26d01" (UID: "ad2a4a21-a0b5-4b90-ae3d-c29e89c26d01"). InnerVolumeSpecName "kube-api-access-9hg8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:27:11.833187 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:27:11.833151    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad2a4a21-a0b5-4b90-ae3d-c29e89c26d01-extensions-socket-volume\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:27:11.833187 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:27:11.833181    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hg8d\" (UniqueName: \"kubernetes.io/projected/ad2a4a21-a0b5-4b90-ae3d-c29e89c26d01-kube-api-access-9hg8d\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:27:12.622605 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:27:12.622573    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q6s9l"
Jul 28 22:27:12.643406 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:27:12.643373    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q6s9l"]
Jul 28 22:27:12.648831 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:27:12.648806    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q6s9l"]
Jul 28 22:27:13.129871 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:27:13.129841    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad2a4a21-a0b5-4b90-ae3d-c29e89c26d01" path="/var/lib/kubelet/pods/ad2a4a21-a0b5-4b90-ae3d-c29e89c26d01/volumes"
Jul 28 22:28:03.143204 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:28:03.143169    2563 scope.go:117] "RemoveContainer" containerID="94a5462239650c437ad5fe18f3cc20efdacb9436edefad8e07bb169030716d48"
Jul 28 22:28:10.050904 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:28:10.050862    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6hfw5"]
Jul 28 22:28:10.051389 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:28:10.051197    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad2a4a21-a0b5-4b90-ae3d-c29e89c26d01" containerName="manager"
Jul 28 22:28:10.051389 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:28:10.051207    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad2a4a21-a0b5-4b90-ae3d-c29e89c26d01" containerName="manager"
Jul 28 22:28:10.051389 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:28:10.051292    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad2a4a21-a0b5-4b90-ae3d-c29e89c26d01" containerName="manager"
Jul 28 22:28:10.055853 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:28:10.055834    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6hfw5"
Jul 28 22:28:10.059048 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:28:10.059021    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-pknb7\""
Jul 28 22:28:10.071630 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:28:10.071605    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6hfw5"]
Jul 28 22:28:10.121019 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:28:10.120987    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7eb29594-0911-45bb-bf64-52eb49612f7d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6hfw5\" (UID: \"7eb29594-0911-45bb-bf64-52eb49612f7d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6hfw5"
Jul 28 22:28:10.121156 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:28:10.121043    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wghg\" (UniqueName: \"kubernetes.io/projected/7eb29594-0911-45bb-bf64-52eb49612f7d-kube-api-access-5wghg\") pod \"kuadrant-operator-controller-manager-86878c769f-6hfw5\" (UID: \"7eb29594-0911-45bb-bf64-52eb49612f7d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6hfw5"
Jul 28 22:28:10.221841 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:28:10.221801    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7eb29594-0911-45bb-bf64-52eb49612f7d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6hfw5\" (UID: \"7eb29594-0911-45bb-bf64-52eb49612f7d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6hfw5"
Jul 28 22:28:10.221999 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:28:10.221951    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wghg\" (UniqueName: \"kubernetes.io/projected/7eb29594-0911-45bb-bf64-52eb49612f7d-kube-api-access-5wghg\") pod \"kuadrant-operator-controller-manager-86878c769f-6hfw5\" (UID: \"7eb29594-0911-45bb-bf64-52eb49612f7d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6hfw5"
Jul 28 22:28:10.222236 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:28:10.222214    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7eb29594-0911-45bb-bf64-52eb49612f7d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6hfw5\" (UID: \"7eb29594-0911-45bb-bf64-52eb49612f7d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6hfw5"
Jul 28 22:28:10.231042 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:28:10.231019    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wghg\" (UniqueName: \"kubernetes.io/projected/7eb29594-0911-45bb-bf64-52eb49612f7d-kube-api-access-5wghg\") pod \"kuadrant-operator-controller-manager-86878c769f-6hfw5\" (UID: \"7eb29594-0911-45bb-bf64-52eb49612f7d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6hfw5"
Jul 28 22:28:10.365329 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:28:10.365232    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6hfw5"
Jul 28 22:28:10.506803 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:28:10.501781    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6hfw5"]
Jul 28 22:28:10.885965 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:28:10.885925    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6hfw5" event={"ID":"7eb29594-0911-45bb-bf64-52eb49612f7d","Type":"ContainerStarted","Data":"6917b7f88ea02aaf7db19c4ed9588258c98c164d887508634053b175b320043a"}
Jul 28 22:28:10.886148 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:28:10.885981    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6hfw5" event={"ID":"7eb29594-0911-45bb-bf64-52eb49612f7d","Type":"ContainerStarted","Data":"8485f6d758395861c4ef9dfede17ce665d956e51c63a43448249d21dc9a77d8f"}
Jul 28 22:28:10.886148 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:28:10.886018    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6hfw5"
Jul 28 22:28:10.919287 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:28:10.919239    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6hfw5" podStartSLOduration=0.919224417 podStartE2EDuration="919.224417ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:28:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:28:10.918185273 +0000 UTC m=+1388.298322916" watchObservedRunningTime="2026-07-28 22:28:10.919224417 +0000 UTC m=+1388.299362118"
Jul 28 22:28:21.893063 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:28:21.892985    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6hfw5"
Jul 28 22:30:03.093239 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:30:03.093206    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mcj8p_7bab32b0-13c6-4055-916e-a7454bafb5b6/ovn-acl-logging/0.log"
Jul 28 22:30:03.094190 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:30:03.094162    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mcj8p_7bab32b0-13c6-4055-916e-a7454bafb5b6/ovn-acl-logging/0.log"
Jul 28 22:35:03.124203 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:03.124173    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mcj8p_7bab32b0-13c6-4055-916e-a7454bafb5b6/ovn-acl-logging/0.log"
Jul 28 22:35:03.124780 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:03.124758    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mcj8p_7bab32b0-13c6-4055-916e-a7454bafb5b6/ovn-acl-logging/0.log"
Jul 28 22:35:38.795762 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:38.795729    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7833e2f6-78c79997bf-5gxgs"]
Jul 28 22:35:38.799150 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:38.799130    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7833e2f6-78c79997bf-5gxgs"
Jul 28 22:35:38.801603 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:38.801575    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-7833e2f6-serving-cert\""
Jul 28 22:35:38.810408 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:38.810383    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7833e2f6-78c79997bf-5gxgs"]
Jul 28 22:35:38.846338 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:38.846304    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf269733-ff0e-461b-bb0a-456b370c98a8-maas-api-tls\") pod \"maas-api-e2e-ait-7833e2f6-78c79997bf-5gxgs\" (UID: \"bf269733-ff0e-461b-bb0a-456b370c98a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7833e2f6-78c79997bf-5gxgs"
Jul 28 22:35:38.846338 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:38.846336    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf269733-ff0e-461b-bb0a-456b370c98a8-openshift-service-ca\") pod \"maas-api-e2e-ait-7833e2f6-78c79997bf-5gxgs\" (UID: \"bf269733-ff0e-461b-bb0a-456b370c98a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7833e2f6-78c79997bf-5gxgs"
Jul 28 22:35:38.846545 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:38.846453    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7cfn\" (UniqueName: \"kubernetes.io/projected/bf269733-ff0e-461b-bb0a-456b370c98a8-kube-api-access-v7cfn\") pod \"maas-api-e2e-ait-7833e2f6-78c79997bf-5gxgs\" (UID: \"bf269733-ff0e-461b-bb0a-456b370c98a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7833e2f6-78c79997bf-5gxgs"
Jul 28 22:35:38.947384 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:38.947333    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf269733-ff0e-461b-bb0a-456b370c98a8-maas-api-tls\") pod \"maas-api-e2e-ait-7833e2f6-78c79997bf-5gxgs\" (UID: \"bf269733-ff0e-461b-bb0a-456b370c98a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7833e2f6-78c79997bf-5gxgs"
Jul 28 22:35:38.947384 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:38.947386    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf269733-ff0e-461b-bb0a-456b370c98a8-openshift-service-ca\") pod \"maas-api-e2e-ait-7833e2f6-78c79997bf-5gxgs\" (UID: \"bf269733-ff0e-461b-bb0a-456b370c98a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7833e2f6-78c79997bf-5gxgs"
Jul 28 22:35:38.947648 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:38.947443    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7cfn\" (UniqueName: \"kubernetes.io/projected/bf269733-ff0e-461b-bb0a-456b370c98a8-kube-api-access-v7cfn\") pod \"maas-api-e2e-ait-7833e2f6-78c79997bf-5gxgs\" (UID: \"bf269733-ff0e-461b-bb0a-456b370c98a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7833e2f6-78c79997bf-5gxgs"
Jul 28 22:35:38.947648 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:35:38.947485    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-7833e2f6-serving-cert: secret "maas-api-e2e-ait-7833e2f6-serving-cert" not found
Jul 28 22:35:38.947648 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:35:38.947551    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf269733-ff0e-461b-bb0a-456b370c98a8-maas-api-tls podName:bf269733-ff0e-461b-bb0a-456b370c98a8 nodeName:}" failed. No retries permitted until 2026-07-28 22:35:39.447536225 +0000 UTC m=+1836.827673847 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bf269733-ff0e-461b-bb0a-456b370c98a8-maas-api-tls") pod "maas-api-e2e-ait-7833e2f6-78c79997bf-5gxgs" (UID: "bf269733-ff0e-461b-bb0a-456b370c98a8") : secret "maas-api-e2e-ait-7833e2f6-serving-cert" not found
Jul 28 22:35:38.948028 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:38.948008    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf269733-ff0e-461b-bb0a-456b370c98a8-openshift-service-ca\") pod \"maas-api-e2e-ait-7833e2f6-78c79997bf-5gxgs\" (UID: \"bf269733-ff0e-461b-bb0a-456b370c98a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7833e2f6-78c79997bf-5gxgs"
Jul 28 22:35:38.956477 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:38.956456    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7cfn\" (UniqueName: \"kubernetes.io/projected/bf269733-ff0e-461b-bb0a-456b370c98a8-kube-api-access-v7cfn\") pod \"maas-api-e2e-ait-7833e2f6-78c79997bf-5gxgs\" (UID: \"bf269733-ff0e-461b-bb0a-456b370c98a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7833e2f6-78c79997bf-5gxgs"
Jul 28 22:35:39.452347 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:39.452308    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf269733-ff0e-461b-bb0a-456b370c98a8-maas-api-tls\") pod \"maas-api-e2e-ait-7833e2f6-78c79997bf-5gxgs\" (UID: \"bf269733-ff0e-461b-bb0a-456b370c98a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7833e2f6-78c79997bf-5gxgs"
Jul 28 22:35:39.454743 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:39.454722    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf269733-ff0e-461b-bb0a-456b370c98a8-maas-api-tls\") pod \"maas-api-e2e-ait-7833e2f6-78c79997bf-5gxgs\" (UID: \"bf269733-ff0e-461b-bb0a-456b370c98a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7833e2f6-78c79997bf-5gxgs"
Jul 28 22:35:39.457390 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:39.457344    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7833e2f6-b4c766dd-c24nt"]
Jul 28 22:35:39.460864 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:39.460846    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7833e2f6-b4c766dd-c24nt"
Jul 28 22:35:39.467188 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:39.467166    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-7833e2f6-dockercfg-xwf4r\""
Jul 28 22:35:39.467443 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:39.467421    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-7833e2f6\""
Jul 28 22:35:39.482799 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:39.482777    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7833e2f6-b4c766dd-c24nt"]
Jul 28 22:35:39.501451 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:39.501428    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7833e2f6-7fd8798976-bbgcr"]
Jul 28 22:35:39.504639 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:39.504625    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7833e2f6-7fd8798976-bbgcr"
Jul 28 22:35:39.535207 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:39.535183    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7833e2f6-7fd8798976-bbgcr"]
Jul 28 22:35:39.554011 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:39.553854    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8a870dd-a058-485c-a17a-13d6081e9c76-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7833e2f6-b4c766dd-c24nt\" (UID: \"f8a870dd-a058-485c-a17a-13d6081e9c76\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7833e2f6-b4c766dd-c24nt"
Jul 28 22:35:39.554011 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:39.553922    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85684\" (UniqueName: \"kubernetes.io/projected/f8a870dd-a058-485c-a17a-13d6081e9c76-kube-api-access-85684\") pod \"payload-pre-processing-e2e-ait-7833e2f6-b4c766dd-c24nt\" (UID: \"f8a870dd-a058-485c-a17a-13d6081e9c76\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7833e2f6-b4c766dd-c24nt"
Jul 28 22:35:39.554011 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:39.553970    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f9c429d-e5f4-49e1-a5df-0bd902a1ea74-plugins-config-volume\") pod \"payload-processing-e2e-ait-7833e2f6-7fd8798976-bbgcr\" (UID: \"4f9c429d-e5f4-49e1-a5df-0bd902a1ea74\") " pod="openshift-ingress/payload-processing-e2e-ait-7833e2f6-7fd8798976-bbgcr"
Jul 28 22:35:39.554593 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:39.554103    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz4hs\" (UniqueName: \"kubernetes.io/projected/4f9c429d-e5f4-49e1-a5df-0bd902a1ea74-kube-api-access-jz4hs\") pod \"payload-processing-e2e-ait-7833e2f6-7fd8798976-bbgcr\" (UID: \"4f9c429d-e5f4-49e1-a5df-0bd902a1ea74\") " pod="openshift-ingress/payload-processing-e2e-ait-7833e2f6-7fd8798976-bbgcr"
Jul 28 22:35:39.655180 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:39.655143    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8a870dd-a058-485c-a17a-13d6081e9c76-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7833e2f6-b4c766dd-c24nt\" (UID: \"f8a870dd-a058-485c-a17a-13d6081e9c76\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7833e2f6-b4c766dd-c24nt"
Jul 28 22:35:39.655335 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:39.655191    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85684\" (UniqueName: \"kubernetes.io/projected/f8a870dd-a058-485c-a17a-13d6081e9c76-kube-api-access-85684\") pod \"payload-pre-processing-e2e-ait-7833e2f6-b4c766dd-c24nt\" (UID: \"f8a870dd-a058-485c-a17a-13d6081e9c76\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7833e2f6-b4c766dd-c24nt"
Jul 28 22:35:39.655335 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:39.655210    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f9c429d-e5f4-49e1-a5df-0bd902a1ea74-plugins-config-volume\") pod \"payload-processing-e2e-ait-7833e2f6-7fd8798976-bbgcr\" (UID: \"4f9c429d-e5f4-49e1-a5df-0bd902a1ea74\") " pod="openshift-ingress/payload-processing-e2e-ait-7833e2f6-7fd8798976-bbgcr"
Jul 28 22:35:39.655448 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:39.655396    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz4hs\" (UniqueName: \"kubernetes.io/projected/4f9c429d-e5f4-49e1-a5df-0bd902a1ea74-kube-api-access-jz4hs\") pod \"payload-processing-e2e-ait-7833e2f6-7fd8798976-bbgcr\" (UID: \"4f9c429d-e5f4-49e1-a5df-0bd902a1ea74\") " pod="openshift-ingress/payload-processing-e2e-ait-7833e2f6-7fd8798976-bbgcr"
Jul 28 22:35:39.655987 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:39.655968    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f9c429d-e5f4-49e1-a5df-0bd902a1ea74-plugins-config-volume\") pod \"payload-processing-e2e-ait-7833e2f6-7fd8798976-bbgcr\" (UID: \"4f9c429d-e5f4-49e1-a5df-0bd902a1ea74\") " pod="openshift-ingress/payload-processing-e2e-ait-7833e2f6-7fd8798976-bbgcr"
Jul 28 22:35:39.656032 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:39.655966    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8a870dd-a058-485c-a17a-13d6081e9c76-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7833e2f6-b4c766dd-c24nt\" (UID: \"f8a870dd-a058-485c-a17a-13d6081e9c76\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7833e2f6-b4c766dd-c24nt"
Jul 28 22:35:39.671456 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:39.671430    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85684\" (UniqueName: \"kubernetes.io/projected/f8a870dd-a058-485c-a17a-13d6081e9c76-kube-api-access-85684\") pod \"payload-pre-processing-e2e-ait-7833e2f6-b4c766dd-c24nt\" (UID: \"f8a870dd-a058-485c-a17a-13d6081e9c76\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7833e2f6-b4c766dd-c24nt"
Jul 28 22:35:39.672615 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:39.672595    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz4hs\" (UniqueName: \"kubernetes.io/projected/4f9c429d-e5f4-49e1-a5df-0bd902a1ea74-kube-api-access-jz4hs\") pod \"payload-processing-e2e-ait-7833e2f6-7fd8798976-bbgcr\" (UID: \"4f9c429d-e5f4-49e1-a5df-0bd902a1ea74\") " pod="openshift-ingress/payload-processing-e2e-ait-7833e2f6-7fd8798976-bbgcr"
Jul 28 22:35:39.710492 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:39.710427    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7833e2f6-78c79997bf-5gxgs"
Jul 28 22:35:39.771041 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:39.771007    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7833e2f6-b4c766dd-c24nt"
Jul 28 22:35:39.815946 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:39.815703    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7833e2f6-7fd8798976-bbgcr"
Jul 28 22:35:39.864996 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:39.864968    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7833e2f6-78c79997bf-5gxgs"]
Jul 28 22:35:39.870575 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:39.870304    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 22:35:39.924587 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:39.924549    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7833e2f6-b4c766dd-c24nt"]
Jul 28 22:35:39.927002 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:35:39.926961    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8a870dd_a058_485c_a17a_13d6081e9c76.slice/crio-a311cd8728efa6b93e651f48b518dc48a17f208b704ba29c437f19579de2f36a WatchSource:0}: Error finding container a311cd8728efa6b93e651f48b518dc48a17f208b704ba29c437f19579de2f36a: Status 404 returned error can't find the container with id a311cd8728efa6b93e651f48b518dc48a17f208b704ba29c437f19579de2f36a
Jul 28 22:35:39.953401 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:39.953074    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7833e2f6-b4c766dd-c24nt" event={"ID":"f8a870dd-a058-485c-a17a-13d6081e9c76","Type":"ContainerStarted","Data":"a311cd8728efa6b93e651f48b518dc48a17f208b704ba29c437f19579de2f36a"}
Jul 28 22:35:39.954900 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:39.954875    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7833e2f6-78c79997bf-5gxgs" event={"ID":"bf269733-ff0e-461b-bb0a-456b370c98a8","Type":"ContainerStarted","Data":"327fa78ba6ca206aa3677be3cdfe291a72ca32205392d85a57d40f4bee02b6df"}
Jul 28 22:35:40.002727 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:40.002701    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7833e2f6-7fd8798976-bbgcr"]
Jul 28 22:35:40.003945 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:35:40.003909    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f9c429d_e5f4_49e1_a5df_0bd902a1ea74.slice/crio-e0502776f03c620b438ddfe513d30104565ce66dfab704ae156d9d688dc135ab WatchSource:0}: Error finding container e0502776f03c620b438ddfe513d30104565ce66dfab704ae156d9d688dc135ab: Status 404 returned error can't find the container with id e0502776f03c620b438ddfe513d30104565ce66dfab704ae156d9d688dc135ab
Jul 28 22:35:40.961329 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:40.961283    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7833e2f6-7fd8798976-bbgcr" event={"ID":"4f9c429d-e5f4-49e1-a5df-0bd902a1ea74","Type":"ContainerStarted","Data":"74bc874c8cf1ee493ddb183ea88130766ac478efdfd1d5a2b8d4585d6ff9577b"}
Jul 28 22:35:40.961853 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:40.961380    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-7833e2f6-7fd8798976-bbgcr"
Jul 28 22:35:40.961853 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:40.961410    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7833e2f6-7fd8798976-bbgcr" event={"ID":"4f9c429d-e5f4-49e1-a5df-0bd902a1ea74","Type":"ContainerStarted","Data":"e0502776f03c620b438ddfe513d30104565ce66dfab704ae156d9d688dc135ab"}
Jul 28 22:35:40.962964 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:40.962933    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7833e2f6-b4c766dd-c24nt" event={"ID":"f8a870dd-a058-485c-a17a-13d6081e9c76","Type":"ContainerStarted","Data":"56c90d6997fd34821b8b8aac03f22fb9075cc851a85bb74c1a6451c2390181d8"}
Jul 28 22:35:40.963096 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:40.962989    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7833e2f6-b4c766dd-c24nt"
Jul 28 22:35:40.964486 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:40.964460    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7833e2f6-78c79997bf-5gxgs" event={"ID":"bf269733-ff0e-461b-bb0a-456b370c98a8","Type":"ContainerStarted","Data":"c36a76a0051d2d80aec5eff8581ac0ee5f460b290c158ce27aaf8dea54cef622"}
Jul 28 22:35:40.964599 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:40.964586    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7833e2f6-78c79997bf-5gxgs"
Jul 28 22:35:40.980604 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:40.980556    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-7833e2f6-7fd8798976-bbgcr" podStartSLOduration=1.980535199 podStartE2EDuration="1.980535199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:35:40.979340353 +0000 UTC m=+1838.359477999" watchObservedRunningTime="2026-07-28 22:35:40.980535199 +0000 UTC m=+1838.360672833"
Jul 28 22:35:40.996954 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:40.996910    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7833e2f6-78c79997bf-5gxgs" podStartSLOduration=2.5333152610000003 podStartE2EDuration="2.996895479s" totalImagesPullingTime="463.580218ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:35:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:35:40.996376662 +0000 UTC m=+1838.376514298" watchObservedRunningTime="2026-07-28 22:35:40.996895479 +0000 UTC m=+1838.377033156"
Jul 28 22:35:41.014585 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:41.014539    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-7833e2f6-b4c766dd-c24nt" podStartSLOduration=2.014520762 podStartE2EDuration="2.014520762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:35:41.012808461 +0000 UTC m=+1838.392946129" watchObservedRunningTime="2026-07-28 22:35:41.014520762 +0000 UTC m=+1838.394658406"
Jul 28 22:35:46.975541 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:46.975514    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7833e2f6-78c79997bf-5gxgs"
Jul 28 22:35:51.971136 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:51.971107    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-7833e2f6-7fd8798976-bbgcr"
Jul 28 22:35:51.971622 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:35:51.971182    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7833e2f6-b4c766dd-c24nt"
Jul 28 22:36:09.383904 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:09.383866    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7833e2f6-78c79997bf-5gxgs"]
Jul 28 22:36:09.384447 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:09.384064    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7833e2f6-78c79997bf-5gxgs" podUID="bf269733-ff0e-461b-bb0a-456b370c98a8" containerName="maas-api" containerID="cri-o://c36a76a0051d2d80aec5eff8581ac0ee5f460b290c158ce27aaf8dea54cef622" gracePeriod=30
Jul 28 22:36:09.450258 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:09.450222    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7833e2f6-7fd8798976-bbgcr"]
Jul 28 22:36:09.450530 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:09.450482    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-7833e2f6-7fd8798976-bbgcr" podUID="4f9c429d-e5f4-49e1-a5df-0bd902a1ea74" containerName="payload-processing" containerID="cri-o://74bc874c8cf1ee493ddb183ea88130766ac478efdfd1d5a2b8d4585d6ff9577b" gracePeriod=30
Jul 28 22:36:09.463532 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:09.462951    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7833e2f6-b4c766dd-c24nt"]
Jul 28 22:36:09.463532 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:09.463174    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-7833e2f6-b4c766dd-c24nt" podUID="f8a870dd-a058-485c-a17a-13d6081e9c76" containerName="payload-pre-processing" containerID="cri-o://56c90d6997fd34821b8b8aac03f22fb9075cc851a85bb74c1a6451c2390181d8" gracePeriod=30
Jul 28 22:36:09.736093 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:09.735803    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7833e2f6-78c79997bf-5gxgs"
Jul 28 22:36:09.740423 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:09.740329    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7833e2f6-7fd8798976-bbgcr"
Jul 28 22:36:09.777325 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:09.777297    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7833e2f6-b4c766dd-c24nt"
Jul 28 22:36:09.825094 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:09.825067    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jz4hs\" (UniqueName: \"kubernetes.io/projected/4f9c429d-e5f4-49e1-a5df-0bd902a1ea74-kube-api-access-jz4hs\") pod \"4f9c429d-e5f4-49e1-a5df-0bd902a1ea74\" (UID: \"4f9c429d-e5f4-49e1-a5df-0bd902a1ea74\") "
Jul 28 22:36:09.825280 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:09.825116    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f9c429d-e5f4-49e1-a5df-0bd902a1ea74-plugins-config-volume\") pod \"4f9c429d-e5f4-49e1-a5df-0bd902a1ea74\" (UID: \"4f9c429d-e5f4-49e1-a5df-0bd902a1ea74\") "
Jul 28 22:36:09.825280 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:09.825201    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf269733-ff0e-461b-bb0a-456b370c98a8-maas-api-tls\") pod \"bf269733-ff0e-461b-bb0a-456b370c98a8\" (UID: \"bf269733-ff0e-461b-bb0a-456b370c98a8\") "
Jul 28 22:36:09.825280 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:09.825223    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7cfn\" (UniqueName: \"kubernetes.io/projected/bf269733-ff0e-461b-bb0a-456b370c98a8-kube-api-access-v7cfn\") pod \"bf269733-ff0e-461b-bb0a-456b370c98a8\" (UID: \"bf269733-ff0e-461b-bb0a-456b370c98a8\") "
Jul 28 22:36:09.825280 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:09.825247    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf269733-ff0e-461b-bb0a-456b370c98a8-openshift-service-ca\") pod \"bf269733-ff0e-461b-bb0a-456b370c98a8\" (UID: \"bf269733-ff0e-461b-bb0a-456b370c98a8\") "
Jul 28 22:36:09.825738 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:09.825668    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf269733-ff0e-461b-bb0a-456b370c98a8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bf269733-ff0e-461b-bb0a-456b370c98a8" (UID: "bf269733-ff0e-461b-bb0a-456b370c98a8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:36:09.825888 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:09.825637    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f9c429d-e5f4-49e1-a5df-0bd902a1ea74-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4f9c429d-e5f4-49e1-a5df-0bd902a1ea74" (UID: "4f9c429d-e5f4-49e1-a5df-0bd902a1ea74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:36:09.827676 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:09.827651    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf269733-ff0e-461b-bb0a-456b370c98a8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bf269733-ff0e-461b-bb0a-456b370c98a8" (UID: "bf269733-ff0e-461b-bb0a-456b370c98a8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:36:09.827761 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:09.827687    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f9c429d-e5f4-49e1-a5df-0bd902a1ea74-kube-api-access-jz4hs" (OuterVolumeSpecName: "kube-api-access-jz4hs") pod "4f9c429d-e5f4-49e1-a5df-0bd902a1ea74" (UID: "4f9c429d-e5f4-49e1-a5df-0bd902a1ea74"). InnerVolumeSpecName "kube-api-access-jz4hs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:36:09.827761 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:09.827715    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf269733-ff0e-461b-bb0a-456b370c98a8-kube-api-access-v7cfn" (OuterVolumeSpecName: "kube-api-access-v7cfn") pod "bf269733-ff0e-461b-bb0a-456b370c98a8" (UID: "bf269733-ff0e-461b-bb0a-456b370c98a8"). InnerVolumeSpecName "kube-api-access-v7cfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:36:09.926418 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:09.926315    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8a870dd-a058-485c-a17a-13d6081e9c76-plugins-config-volume\") pod \"f8a870dd-a058-485c-a17a-13d6081e9c76\" (UID: \"f8a870dd-a058-485c-a17a-13d6081e9c76\") "
Jul 28 22:36:09.926418 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:09.926412    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85684\" (UniqueName: \"kubernetes.io/projected/f8a870dd-a058-485c-a17a-13d6081e9c76-kube-api-access-85684\") pod \"f8a870dd-a058-485c-a17a-13d6081e9c76\" (UID: \"f8a870dd-a058-485c-a17a-13d6081e9c76\") "
Jul 28 22:36:09.926608 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:09.926588    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf269733-ff0e-461b-bb0a-456b370c98a8-maas-api-tls\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:36:09.926608 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:09.926600    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7cfn\" (UniqueName: \"kubernetes.io/projected/bf269733-ff0e-461b-bb0a-456b370c98a8-kube-api-access-v7cfn\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:36:09.926712 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:09.926610    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf269733-ff0e-461b-bb0a-456b370c98a8-openshift-service-ca\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:36:09.926712 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:09.926619    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz4hs\" (UniqueName: \"kubernetes.io/projected/4f9c429d-e5f4-49e1-a5df-0bd902a1ea74-kube-api-access-jz4hs\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:36:09.926712 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:09.926627    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f9c429d-e5f4-49e1-a5df-0bd902a1ea74-plugins-config-volume\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:36:09.926801 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:09.926715    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8a870dd-a058-485c-a17a-13d6081e9c76-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f8a870dd-a058-485c-a17a-13d6081e9c76" (UID: "f8a870dd-a058-485c-a17a-13d6081e9c76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:36:09.928386 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:09.928363    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a870dd-a058-485c-a17a-13d6081e9c76-kube-api-access-85684" (OuterVolumeSpecName: "kube-api-access-85684") pod "f8a870dd-a058-485c-a17a-13d6081e9c76" (UID: "f8a870dd-a058-485c-a17a-13d6081e9c76"). InnerVolumeSpecName "kube-api-access-85684". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:36:10.027832 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:10.027792    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85684\" (UniqueName: \"kubernetes.io/projected/f8a870dd-a058-485c-a17a-13d6081e9c76-kube-api-access-85684\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:36:10.027832 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:10.027826    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8a870dd-a058-485c-a17a-13d6081e9c76-plugins-config-volume\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:36:10.102532 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:10.102498    2563 generic.go:356] "Generic (PLEG): container finished" podID="4f9c429d-e5f4-49e1-a5df-0bd902a1ea74" containerID="74bc874c8cf1ee493ddb183ea88130766ac478efdfd1d5a2b8d4585d6ff9577b" exitCode=0
Jul 28 22:36:10.102709 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:10.102540    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7833e2f6-7fd8798976-bbgcr"
Jul 28 22:36:10.102709 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:10.102580    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7833e2f6-7fd8798976-bbgcr" event={"ID":"4f9c429d-e5f4-49e1-a5df-0bd902a1ea74","Type":"ContainerDied","Data":"74bc874c8cf1ee493ddb183ea88130766ac478efdfd1d5a2b8d4585d6ff9577b"}
Jul 28 22:36:10.102709 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:10.102642    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7833e2f6-7fd8798976-bbgcr" event={"ID":"4f9c429d-e5f4-49e1-a5df-0bd902a1ea74","Type":"ContainerDied","Data":"e0502776f03c620b438ddfe513d30104565ce66dfab704ae156d9d688dc135ab"}
Jul 28 22:36:10.102709 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:10.102671    2563 scope.go:117] "RemoveContainer" containerID="74bc874c8cf1ee493ddb183ea88130766ac478efdfd1d5a2b8d4585d6ff9577b"
Jul 28 22:36:10.103830 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:10.103811    2563 generic.go:356] "Generic (PLEG): container finished" podID="f8a870dd-a058-485c-a17a-13d6081e9c76" containerID="56c90d6997fd34821b8b8aac03f22fb9075cc851a85bb74c1a6451c2390181d8" exitCode=0
Jul 28 22:36:10.103896 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:10.103871    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7833e2f6-b4c766dd-c24nt"
Jul 28 22:36:10.103953 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:10.103876    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7833e2f6-b4c766dd-c24nt" event={"ID":"f8a870dd-a058-485c-a17a-13d6081e9c76","Type":"ContainerDied","Data":"56c90d6997fd34821b8b8aac03f22fb9075cc851a85bb74c1a6451c2390181d8"}
Jul 28 22:36:10.104009 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:10.103996    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7833e2f6-b4c766dd-c24nt" event={"ID":"f8a870dd-a058-485c-a17a-13d6081e9c76","Type":"ContainerDied","Data":"a311cd8728efa6b93e651f48b518dc48a17f208b704ba29c437f19579de2f36a"}
Jul 28 22:36:10.105219 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:10.105194    2563 generic.go:356] "Generic (PLEG): container finished" podID="bf269733-ff0e-461b-bb0a-456b370c98a8" containerID="c36a76a0051d2d80aec5eff8581ac0ee5f460b290c158ce27aaf8dea54cef622" exitCode=0
Jul 28 22:36:10.105219 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:10.105231    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7833e2f6-78c79997bf-5gxgs"
Jul 28 22:36:10.105436 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:10.105261    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7833e2f6-78c79997bf-5gxgs" event={"ID":"bf269733-ff0e-461b-bb0a-456b370c98a8","Type":"ContainerDied","Data":"c36a76a0051d2d80aec5eff8581ac0ee5f460b290c158ce27aaf8dea54cef622"}
Jul 28 22:36:10.105436 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:10.105302    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7833e2f6-78c79997bf-5gxgs" event={"ID":"bf269733-ff0e-461b-bb0a-456b370c98a8","Type":"ContainerDied","Data":"327fa78ba6ca206aa3677be3cdfe291a72ca32205392d85a57d40f4bee02b6df"}
Jul 28 22:36:10.120070 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:10.120048    2563 scope.go:117] "RemoveContainer" containerID="74bc874c8cf1ee493ddb183ea88130766ac478efdfd1d5a2b8d4585d6ff9577b"
Jul 28 22:36:10.120395 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:36:10.120367    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74bc874c8cf1ee493ddb183ea88130766ac478efdfd1d5a2b8d4585d6ff9577b\": container with ID starting with 74bc874c8cf1ee493ddb183ea88130766ac478efdfd1d5a2b8d4585d6ff9577b not found: ID does not exist" containerID="74bc874c8cf1ee493ddb183ea88130766ac478efdfd1d5a2b8d4585d6ff9577b"
Jul 28 22:36:10.120490 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:10.120405    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74bc874c8cf1ee493ddb183ea88130766ac478efdfd1d5a2b8d4585d6ff9577b"} err="failed to get container status \"74bc874c8cf1ee493ddb183ea88130766ac478efdfd1d5a2b8d4585d6ff9577b\": rpc error: code = NotFound desc = could not find container \"74bc874c8cf1ee493ddb183ea88130766ac478efdfd1d5a2b8d4585d6ff9577b\": container with ID starting with 74bc874c8cf1ee493ddb183ea88130766ac478efdfd1d5a2b8d4585d6ff9577b not found: ID does not exist"
Jul 28 22:36:10.120490 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:10.120428    2563 scope.go:117] "RemoveContainer" containerID="56c90d6997fd34821b8b8aac03f22fb9075cc851a85bb74c1a6451c2390181d8"
Jul 28 22:36:10.128933 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:10.128913    2563 scope.go:117] "RemoveContainer" containerID="56c90d6997fd34821b8b8aac03f22fb9075cc851a85bb74c1a6451c2390181d8"
Jul 28 22:36:10.129167 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:36:10.129149    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56c90d6997fd34821b8b8aac03f22fb9075cc851a85bb74c1a6451c2390181d8\": container with ID starting with 56c90d6997fd34821b8b8aac03f22fb9075cc851a85bb74c1a6451c2390181d8 not found: ID does not exist" containerID="56c90d6997fd34821b8b8aac03f22fb9075cc851a85bb74c1a6451c2390181d8"
Jul 28 22:36:10.129214 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:10.129175    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56c90d6997fd34821b8b8aac03f22fb9075cc851a85bb74c1a6451c2390181d8"} err="failed to get container status \"56c90d6997fd34821b8b8aac03f22fb9075cc851a85bb74c1a6451c2390181d8\": rpc error: code = NotFound desc = could not find container \"56c90d6997fd34821b8b8aac03f22fb9075cc851a85bb74c1a6451c2390181d8\": container with ID starting with 56c90d6997fd34821b8b8aac03f22fb9075cc851a85bb74c1a6451c2390181d8 not found: ID does not exist"
Jul 28 22:36:10.129214 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:10.129191    2563 scope.go:117] "RemoveContainer" containerID="c36a76a0051d2d80aec5eff8581ac0ee5f460b290c158ce27aaf8dea54cef622"
Jul 28 22:36:10.137247 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:10.137230    2563 scope.go:117] "RemoveContainer" containerID="c36a76a0051d2d80aec5eff8581ac0ee5f460b290c158ce27aaf8dea54cef622"
Jul 28 22:36:10.137524 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:36:10.137502    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c36a76a0051d2d80aec5eff8581ac0ee5f460b290c158ce27aaf8dea54cef622\": container with ID starting with c36a76a0051d2d80aec5eff8581ac0ee5f460b290c158ce27aaf8dea54cef622 not found: ID does not exist" containerID="c36a76a0051d2d80aec5eff8581ac0ee5f460b290c158ce27aaf8dea54cef622"
Jul 28 22:36:10.137593 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:10.137535    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c36a76a0051d2d80aec5eff8581ac0ee5f460b290c158ce27aaf8dea54cef622"} err="failed to get container status \"c36a76a0051d2d80aec5eff8581ac0ee5f460b290c158ce27aaf8dea54cef622\": rpc error: code = NotFound desc = could not find container \"c36a76a0051d2d80aec5eff8581ac0ee5f460b290c158ce27aaf8dea54cef622\": container with ID starting with c36a76a0051d2d80aec5eff8581ac0ee5f460b290c158ce27aaf8dea54cef622 not found: ID does not exist"
Jul 28 22:36:10.156241 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:10.156220    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7833e2f6-78c79997bf-5gxgs"]
Jul 28 22:36:10.169935 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:10.169913    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7833e2f6-78c79997bf-5gxgs"]
Jul 28 22:36:10.195767 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:10.195748    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7833e2f6-7fd8798976-bbgcr"]
Jul 28 22:36:10.201897 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:10.201853    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7833e2f6-7fd8798976-bbgcr"]
Jul 28 22:36:10.236791 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:10.236765    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7833e2f6-b4c766dd-c24nt"]
Jul 28 22:36:10.247395 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:10.247374    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7833e2f6-b4c766dd-c24nt"]
Jul 28 22:36:11.130052 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:11.130020    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f9c429d-e5f4-49e1-a5df-0bd902a1ea74" path="/var/lib/kubelet/pods/4f9c429d-e5f4-49e1-a5df-0bd902a1ea74/volumes"
Jul 28 22:36:11.130435 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:11.130419    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf269733-ff0e-461b-bb0a-456b370c98a8" path="/var/lib/kubelet/pods/bf269733-ff0e-461b-bb0a-456b370c98a8/volumes"
Jul 28 22:36:11.130737 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:11.130724    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8a870dd-a058-485c-a17a-13d6081e9c76" path="/var/lib/kubelet/pods/f8a870dd-a058-485c-a17a-13d6081e9c76/volumes"
Jul 28 22:36:13.133099 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:13.133058    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-72a5477c-65bfd47b98-l9h9n"]
Jul 28 22:36:13.133607 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:13.133588    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf269733-ff0e-461b-bb0a-456b370c98a8" containerName="maas-api"
Jul 28 22:36:13.133671 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:13.133611    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf269733-ff0e-461b-bb0a-456b370c98a8" containerName="maas-api"
Jul 28 22:36:13.133671 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:13.133652    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8a870dd-a058-485c-a17a-13d6081e9c76" containerName="payload-pre-processing"
Jul 28 22:36:13.133671 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:13.133665    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8a870dd-a058-485c-a17a-13d6081e9c76" containerName="payload-pre-processing"
Jul 28 22:36:13.133776 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:13.133686    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f9c429d-e5f4-49e1-a5df-0bd902a1ea74" containerName="payload-processing"
Jul 28 22:36:13.133776 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:13.133696    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f9c429d-e5f4-49e1-a5df-0bd902a1ea74" containerName="payload-processing"
Jul 28 22:36:13.133839 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:13.133784    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f9c429d-e5f4-49e1-a5df-0bd902a1ea74" containerName="payload-processing"
Jul 28 22:36:13.133839 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:13.133794    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf269733-ff0e-461b-bb0a-456b370c98a8" containerName="maas-api"
Jul 28 22:36:13.133839 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:13.133805    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8a870dd-a058-485c-a17a-13d6081e9c76" containerName="payload-pre-processing"
Jul 28 22:36:13.138685 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:13.138663    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72a5477c-65bfd47b98-l9h9n"
Jul 28 22:36:13.142464 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:13.142445    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-72a5477c-serving-cert\""
Jul 28 22:36:13.147496 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:13.147474    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-72a5477c-65bfd47b98-l9h9n"]
Jul 28 22:36:13.257913 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:13.257877    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppsgm\" (UniqueName: \"kubernetes.io/projected/5c34c245-30af-46c7-a42e-eddec981cd64-kube-api-access-ppsgm\") pod \"maas-api-e2e-migrate-72a5477c-65bfd47b98-l9h9n\" (UID: \"5c34c245-30af-46c7-a42e-eddec981cd64\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72a5477c-65bfd47b98-l9h9n"
Jul 28 22:36:13.258100 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:13.257965    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c34c245-30af-46c7-a42e-eddec981cd64-openshift-service-ca\") pod \"maas-api-e2e-migrate-72a5477c-65bfd47b98-l9h9n\" (UID: \"5c34c245-30af-46c7-a42e-eddec981cd64\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72a5477c-65bfd47b98-l9h9n"
Jul 28 22:36:13.258100 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:13.257997    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c34c245-30af-46c7-a42e-eddec981cd64-maas-api-tls\") pod \"maas-api-e2e-migrate-72a5477c-65bfd47b98-l9h9n\" (UID: \"5c34c245-30af-46c7-a42e-eddec981cd64\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72a5477c-65bfd47b98-l9h9n"
Jul 28 22:36:13.358885 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:13.358845    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppsgm\" (UniqueName: \"kubernetes.io/projected/5c34c245-30af-46c7-a42e-eddec981cd64-kube-api-access-ppsgm\") pod \"maas-api-e2e-migrate-72a5477c-65bfd47b98-l9h9n\" (UID: \"5c34c245-30af-46c7-a42e-eddec981cd64\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72a5477c-65bfd47b98-l9h9n"
Jul 28 22:36:13.359070 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:13.358907    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c34c245-30af-46c7-a42e-eddec981cd64-openshift-service-ca\") pod \"maas-api-e2e-migrate-72a5477c-65bfd47b98-l9h9n\" (UID: \"5c34c245-30af-46c7-a42e-eddec981cd64\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72a5477c-65bfd47b98-l9h9n"
Jul 28 22:36:13.359070 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:13.358926    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c34c245-30af-46c7-a42e-eddec981cd64-maas-api-tls\") pod \"maas-api-e2e-migrate-72a5477c-65bfd47b98-l9h9n\" (UID: \"5c34c245-30af-46c7-a42e-eddec981cd64\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72a5477c-65bfd47b98-l9h9n"
Jul 28 22:36:13.359185 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:36:13.359115    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-72a5477c-serving-cert: secret "maas-api-e2e-migrate-72a5477c-serving-cert" not found
Jul 28 22:36:13.359185 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:36:13.359176    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c34c245-30af-46c7-a42e-eddec981cd64-maas-api-tls podName:5c34c245-30af-46c7-a42e-eddec981cd64 nodeName:}" failed. No retries permitted until 2026-07-28 22:36:13.859159836 +0000 UTC m=+1871.239297459 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5c34c245-30af-46c7-a42e-eddec981cd64-maas-api-tls") pod "maas-api-e2e-migrate-72a5477c-65bfd47b98-l9h9n" (UID: "5c34c245-30af-46c7-a42e-eddec981cd64") : secret "maas-api-e2e-migrate-72a5477c-serving-cert" not found
Jul 28 22:36:13.359702 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:13.359679    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c34c245-30af-46c7-a42e-eddec981cd64-openshift-service-ca\") pod \"maas-api-e2e-migrate-72a5477c-65bfd47b98-l9h9n\" (UID: \"5c34c245-30af-46c7-a42e-eddec981cd64\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72a5477c-65bfd47b98-l9h9n"
Jul 28 22:36:13.368292 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:13.368268    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppsgm\" (UniqueName: \"kubernetes.io/projected/5c34c245-30af-46c7-a42e-eddec981cd64-kube-api-access-ppsgm\") pod \"maas-api-e2e-migrate-72a5477c-65bfd47b98-l9h9n\" (UID: \"5c34c245-30af-46c7-a42e-eddec981cd64\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72a5477c-65bfd47b98-l9h9n"
Jul 28 22:36:13.821218 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:13.821180    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-72a5477c-8f8d88dfc-6mbhz"]
Jul 28 22:36:13.824711 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:13.824687    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-72a5477c-8f8d88dfc-6mbhz"
Jul 28 22:36:13.827701 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:13.827684    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-72a5477c-dockercfg-7szm8\""
Jul 28 22:36:13.828103 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:13.828087    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-72a5477c\""
Jul 28 22:36:13.838282 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:13.838259    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-72a5477c-8f8d88dfc-6mbhz"]
Jul 28 22:36:13.863761 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:13.863738    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c34c245-30af-46c7-a42e-eddec981cd64-maas-api-tls\") pod \"maas-api-e2e-migrate-72a5477c-65bfd47b98-l9h9n\" (UID: \"5c34c245-30af-46c7-a42e-eddec981cd64\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72a5477c-65bfd47b98-l9h9n"
Jul 28 22:36:13.866053 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:13.866026    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c34c245-30af-46c7-a42e-eddec981cd64-maas-api-tls\") pod \"maas-api-e2e-migrate-72a5477c-65bfd47b98-l9h9n\" (UID: \"5c34c245-30af-46c7-a42e-eddec981cd64\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72a5477c-65bfd47b98-l9h9n"
Jul 28 22:36:13.964707 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:13.964667    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl2wk\" (UniqueName: \"kubernetes.io/projected/949e3a69-a086-4c74-bd35-12ee72dcd62f-kube-api-access-vl2wk\") pod \"payload-processing-e2e-migrate-72a5477c-8f8d88dfc-6mbhz\" (UID: \"949e3a69-a086-4c74-bd35-12ee72dcd62f\") " pod="openshift-ingress/payload-processing-e2e-migrate-72a5477c-8f8d88dfc-6mbhz"
Jul 28 22:36:13.964860 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:13.964817    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/949e3a69-a086-4c74-bd35-12ee72dcd62f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-72a5477c-8f8d88dfc-6mbhz\" (UID: \"949e3a69-a086-4c74-bd35-12ee72dcd62f\") " pod="openshift-ingress/payload-processing-e2e-migrate-72a5477c-8f8d88dfc-6mbhz"
Jul 28 22:36:14.051026 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:14.050987    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72a5477c-65bfd47b98-l9h9n"
Jul 28 22:36:14.065278 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:14.065246    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/949e3a69-a086-4c74-bd35-12ee72dcd62f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-72a5477c-8f8d88dfc-6mbhz\" (UID: \"949e3a69-a086-4c74-bd35-12ee72dcd62f\") " pod="openshift-ingress/payload-processing-e2e-migrate-72a5477c-8f8d88dfc-6mbhz"
Jul 28 22:36:14.065410 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:14.065318    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl2wk\" (UniqueName: \"kubernetes.io/projected/949e3a69-a086-4c74-bd35-12ee72dcd62f-kube-api-access-vl2wk\") pod \"payload-processing-e2e-migrate-72a5477c-8f8d88dfc-6mbhz\" (UID: \"949e3a69-a086-4c74-bd35-12ee72dcd62f\") " pod="openshift-ingress/payload-processing-e2e-migrate-72a5477c-8f8d88dfc-6mbhz"
Jul 28 22:36:14.066272 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:14.066246    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/949e3a69-a086-4c74-bd35-12ee72dcd62f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-72a5477c-8f8d88dfc-6mbhz\" (UID: \"949e3a69-a086-4c74-bd35-12ee72dcd62f\") " pod="openshift-ingress/payload-processing-e2e-migrate-72a5477c-8f8d88dfc-6mbhz"
Jul 28 22:36:14.074570 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:14.074498    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl2wk\" (UniqueName: \"kubernetes.io/projected/949e3a69-a086-4c74-bd35-12ee72dcd62f-kube-api-access-vl2wk\") pod \"payload-processing-e2e-migrate-72a5477c-8f8d88dfc-6mbhz\" (UID: \"949e3a69-a086-4c74-bd35-12ee72dcd62f\") " pod="openshift-ingress/payload-processing-e2e-migrate-72a5477c-8f8d88dfc-6mbhz"
Jul 28 22:36:14.144448 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:14.144154    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-72a5477c-8f8d88dfc-6mbhz"
Jul 28 22:36:14.199152 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:14.199049    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-72a5477c-65bfd47b98-l9h9n"]
Jul 28 22:36:14.202549 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:36:14.202515    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c34c245_30af_46c7_a42e_eddec981cd64.slice/crio-fd490c633f2f890990c2053d4582489bde9820711f436fbcec50f6149dccc131 WatchSource:0}: Error finding container fd490c633f2f890990c2053d4582489bde9820711f436fbcec50f6149dccc131: Status 404 returned error can't find the container with id fd490c633f2f890990c2053d4582489bde9820711f436fbcec50f6149dccc131
Jul 28 22:36:14.300956 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:14.300927    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-72a5477c-8f8d88dfc-6mbhz"]
Jul 28 22:36:14.303273 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:36:14.303235    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod949e3a69_a086_4c74_bd35_12ee72dcd62f.slice/crio-b314cc9f8f7bc1e7338534816305f00b1ba5f50fbb80b0ff3e90578d80316f22 WatchSource:0}: Error finding container b314cc9f8f7bc1e7338534816305f00b1ba5f50fbb80b0ff3e90578d80316f22: Status 404 returned error can't find the container with id b314cc9f8f7bc1e7338534816305f00b1ba5f50fbb80b0ff3e90578d80316f22
Jul 28 22:36:15.150008 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:15.149962    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72a5477c-65bfd47b98-l9h9n" event={"ID":"5c34c245-30af-46c7-a42e-eddec981cd64","Type":"ContainerStarted","Data":"151e6711453e0bfb9d0edd3762b833ee15a2662e4c677cf6b4f79082c156233a"}
Jul 28 22:36:15.150621 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:15.150036    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72a5477c-65bfd47b98-l9h9n" event={"ID":"5c34c245-30af-46c7-a42e-eddec981cd64","Type":"ContainerStarted","Data":"fd490c633f2f890990c2053d4582489bde9820711f436fbcec50f6149dccc131"}
Jul 28 22:36:15.150621 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:15.150172    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72a5477c-65bfd47b98-l9h9n"
Jul 28 22:36:15.151700 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:15.151676    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-72a5477c-8f8d88dfc-6mbhz" event={"ID":"949e3a69-a086-4c74-bd35-12ee72dcd62f","Type":"ContainerStarted","Data":"b7d00c1aeb598fc6b68911169b54d2a5bcd26609bf43f11ef8aed7be2628625d"}
Jul 28 22:36:15.151829 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:15.151714    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-72a5477c-8f8d88dfc-6mbhz" event={"ID":"949e3a69-a086-4c74-bd35-12ee72dcd62f","Type":"ContainerStarted","Data":"b314cc9f8f7bc1e7338534816305f00b1ba5f50fbb80b0ff3e90578d80316f22"}
Jul 28 22:36:15.151881 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:15.151824    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-72a5477c-8f8d88dfc-6mbhz"
Jul 28 22:36:15.167114 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:15.167055    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72a5477c-65bfd47b98-l9h9n" podStartSLOduration=1.790765855 podStartE2EDuration="2.167042284s" totalImagesPullingTime="376.276429ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:36:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:36:15.165953467 +0000 UTC m=+1872.546091111" watchObservedRunningTime="2026-07-28 22:36:15.167042284 +0000 UTC m=+1872.547179927"
Jul 28 22:36:15.185638 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:15.185595    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-72a5477c-8f8d88dfc-6mbhz" podStartSLOduration=2.185582236 podStartE2EDuration="2.185582236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:36:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:36:15.1843894 +0000 UTC m=+1872.564527045" watchObservedRunningTime="2026-07-28 22:36:15.185582236 +0000 UTC m=+1872.565719881"
Jul 28 22:36:21.162470 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:21.162437    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72a5477c-65bfd47b98-l9h9n"
Jul 28 22:36:26.158526 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:26.158437    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-72a5477c-8f8d88dfc-6mbhz"
Jul 28 22:36:44.580089 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:44.580049    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-72a5477c-65bfd47b98-l9h9n"]
Jul 28 22:36:44.580563 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:44.580364    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72a5477c-65bfd47b98-l9h9n" podUID="5c34c245-30af-46c7-a42e-eddec981cd64" containerName="maas-api" containerID="cri-o://151e6711453e0bfb9d0edd3762b833ee15a2662e4c677cf6b4f79082c156233a" gracePeriod=30
Jul 28 22:36:44.633230 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:44.633197    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-72a5477c-8f8d88dfc-6mbhz"]
Jul 28 22:36:44.633837 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:44.633518    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-72a5477c-8f8d88dfc-6mbhz" podUID="949e3a69-a086-4c74-bd35-12ee72dcd62f" containerName="payload-processing" containerID="cri-o://b7d00c1aeb598fc6b68911169b54d2a5bcd26609bf43f11ef8aed7be2628625d" gracePeriod=30
Jul 28 22:36:44.882985 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:44.882960    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72a5477c-65bfd47b98-l9h9n"
Jul 28 22:36:44.915117 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:44.915085    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-72a5477c-8f8d88dfc-6mbhz"
Jul 28 22:36:44.931105 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:44.931076    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppsgm\" (UniqueName: \"kubernetes.io/projected/5c34c245-30af-46c7-a42e-eddec981cd64-kube-api-access-ppsgm\") pod \"5c34c245-30af-46c7-a42e-eddec981cd64\" (UID: \"5c34c245-30af-46c7-a42e-eddec981cd64\") "
Jul 28 22:36:44.931968 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:44.931174    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c34c245-30af-46c7-a42e-eddec981cd64-maas-api-tls\") pod \"5c34c245-30af-46c7-a42e-eddec981cd64\" (UID: \"5c34c245-30af-46c7-a42e-eddec981cd64\") "
Jul 28 22:36:44.931968 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:44.931205    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c34c245-30af-46c7-a42e-eddec981cd64-openshift-service-ca\") pod \"5c34c245-30af-46c7-a42e-eddec981cd64\" (UID: \"5c34c245-30af-46c7-a42e-eddec981cd64\") "
Jul 28 22:36:44.935446 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:44.933652    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c34c245-30af-46c7-a42e-eddec981cd64-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5c34c245-30af-46c7-a42e-eddec981cd64" (UID: "5c34c245-30af-46c7-a42e-eddec981cd64"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:36:44.936449 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:44.936387    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c34c245-30af-46c7-a42e-eddec981cd64-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5c34c245-30af-46c7-a42e-eddec981cd64" (UID: "5c34c245-30af-46c7-a42e-eddec981cd64"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:36:44.936585 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:44.936460    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c34c245-30af-46c7-a42e-eddec981cd64-kube-api-access-ppsgm" (OuterVolumeSpecName: "kube-api-access-ppsgm") pod "5c34c245-30af-46c7-a42e-eddec981cd64" (UID: "5c34c245-30af-46c7-a42e-eddec981cd64"). InnerVolumeSpecName "kube-api-access-ppsgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:36:45.032256 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:45.032225    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vl2wk\" (UniqueName: \"kubernetes.io/projected/949e3a69-a086-4c74-bd35-12ee72dcd62f-kube-api-access-vl2wk\") pod \"949e3a69-a086-4c74-bd35-12ee72dcd62f\" (UID: \"949e3a69-a086-4c74-bd35-12ee72dcd62f\") "
Jul 28 22:36:45.032482 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:45.032285    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/949e3a69-a086-4c74-bd35-12ee72dcd62f-plugins-config-volume\") pod \"949e3a69-a086-4c74-bd35-12ee72dcd62f\" (UID: \"949e3a69-a086-4c74-bd35-12ee72dcd62f\") "
Jul 28 22:36:45.032563 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:45.032499    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c34c245-30af-46c7-a42e-eddec981cd64-maas-api-tls\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:36:45.032563 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:45.032519    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c34c245-30af-46c7-a42e-eddec981cd64-openshift-service-ca\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:36:45.032563 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:45.032535    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppsgm\" (UniqueName: \"kubernetes.io/projected/5c34c245-30af-46c7-a42e-eddec981cd64-kube-api-access-ppsgm\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:36:45.032755 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:45.032727    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/949e3a69-a086-4c74-bd35-12ee72dcd62f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "949e3a69-a086-4c74-bd35-12ee72dcd62f" (UID: "949e3a69-a086-4c74-bd35-12ee72dcd62f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:36:45.034196 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:45.034176    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/949e3a69-a086-4c74-bd35-12ee72dcd62f-kube-api-access-vl2wk" (OuterVolumeSpecName: "kube-api-access-vl2wk") pod "949e3a69-a086-4c74-bd35-12ee72dcd62f" (UID: "949e3a69-a086-4c74-bd35-12ee72dcd62f"). InnerVolumeSpecName "kube-api-access-vl2wk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:36:45.133096 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:45.133024    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/949e3a69-a086-4c74-bd35-12ee72dcd62f-plugins-config-volume\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:36:45.133096 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:45.133048    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl2wk\" (UniqueName: \"kubernetes.io/projected/949e3a69-a086-4c74-bd35-12ee72dcd62f-kube-api-access-vl2wk\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:36:45.298860 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:45.298824    2563 generic.go:356] "Generic (PLEG): container finished" podID="5c34c245-30af-46c7-a42e-eddec981cd64" containerID="151e6711453e0bfb9d0edd3762b833ee15a2662e4c677cf6b4f79082c156233a" exitCode=0
Jul 28 22:36:45.299048 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:45.298872    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72a5477c-65bfd47b98-l9h9n"
Jul 28 22:36:45.299048 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:45.298912    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72a5477c-65bfd47b98-l9h9n" event={"ID":"5c34c245-30af-46c7-a42e-eddec981cd64","Type":"ContainerDied","Data":"151e6711453e0bfb9d0edd3762b833ee15a2662e4c677cf6b4f79082c156233a"}
Jul 28 22:36:45.299048 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:45.298970    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-72a5477c-65bfd47b98-l9h9n" event={"ID":"5c34c245-30af-46c7-a42e-eddec981cd64","Type":"ContainerDied","Data":"fd490c633f2f890990c2053d4582489bde9820711f436fbcec50f6149dccc131"}
Jul 28 22:36:45.299048 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:45.298996    2563 scope.go:117] "RemoveContainer" containerID="151e6711453e0bfb9d0edd3762b833ee15a2662e4c677cf6b4f79082c156233a"
Jul 28 22:36:45.300178 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:45.300156    2563 generic.go:356] "Generic (PLEG): container finished" podID="949e3a69-a086-4c74-bd35-12ee72dcd62f" containerID="b7d00c1aeb598fc6b68911169b54d2a5bcd26609bf43f11ef8aed7be2628625d" exitCode=0
Jul 28 22:36:45.300267 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:45.300195    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-72a5477c-8f8d88dfc-6mbhz"
Jul 28 22:36:45.300267 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:45.300209    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-72a5477c-8f8d88dfc-6mbhz" event={"ID":"949e3a69-a086-4c74-bd35-12ee72dcd62f","Type":"ContainerDied","Data":"b7d00c1aeb598fc6b68911169b54d2a5bcd26609bf43f11ef8aed7be2628625d"}
Jul 28 22:36:45.300267 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:45.300246    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-72a5477c-8f8d88dfc-6mbhz" event={"ID":"949e3a69-a086-4c74-bd35-12ee72dcd62f","Type":"ContainerDied","Data":"b314cc9f8f7bc1e7338534816305f00b1ba5f50fbb80b0ff3e90578d80316f22"}
Jul 28 22:36:45.307948 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:45.307927    2563 scope.go:117] "RemoveContainer" containerID="151e6711453e0bfb9d0edd3762b833ee15a2662e4c677cf6b4f79082c156233a"
Jul 28 22:36:45.308231 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:36:45.308212    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"151e6711453e0bfb9d0edd3762b833ee15a2662e4c677cf6b4f79082c156233a\": container with ID starting with 151e6711453e0bfb9d0edd3762b833ee15a2662e4c677cf6b4f79082c156233a not found: ID does not exist" containerID="151e6711453e0bfb9d0edd3762b833ee15a2662e4c677cf6b4f79082c156233a"
Jul 28 22:36:45.308284 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:45.308238    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"151e6711453e0bfb9d0edd3762b833ee15a2662e4c677cf6b4f79082c156233a"} err="failed to get container status \"151e6711453e0bfb9d0edd3762b833ee15a2662e4c677cf6b4f79082c156233a\": rpc error: code = NotFound desc = could not find container \"151e6711453e0bfb9d0edd3762b833ee15a2662e4c677cf6b4f79082c156233a\": container with ID starting with 151e6711453e0bfb9d0edd3762b833ee15a2662e4c677cf6b4f79082c156233a not found: ID does not exist"
Jul 28 22:36:45.308284 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:45.308255    2563 scope.go:117] "RemoveContainer" containerID="b7d00c1aeb598fc6b68911169b54d2a5bcd26609bf43f11ef8aed7be2628625d"
Jul 28 22:36:45.315394 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:45.315378    2563 scope.go:117] "RemoveContainer" containerID="b7d00c1aeb598fc6b68911169b54d2a5bcd26609bf43f11ef8aed7be2628625d"
Jul 28 22:36:45.315628 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:36:45.315612    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7d00c1aeb598fc6b68911169b54d2a5bcd26609bf43f11ef8aed7be2628625d\": container with ID starting with b7d00c1aeb598fc6b68911169b54d2a5bcd26609bf43f11ef8aed7be2628625d not found: ID does not exist" containerID="b7d00c1aeb598fc6b68911169b54d2a5bcd26609bf43f11ef8aed7be2628625d"
Jul 28 22:36:45.315671 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:45.315633    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7d00c1aeb598fc6b68911169b54d2a5bcd26609bf43f11ef8aed7be2628625d"} err="failed to get container status \"b7d00c1aeb598fc6b68911169b54d2a5bcd26609bf43f11ef8aed7be2628625d\": rpc error: code = NotFound desc = could not find container \"b7d00c1aeb598fc6b68911169b54d2a5bcd26609bf43f11ef8aed7be2628625d\": container with ID starting with b7d00c1aeb598fc6b68911169b54d2a5bcd26609bf43f11ef8aed7be2628625d not found: ID does not exist"
Jul 28 22:36:45.318229 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:45.318209    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-72a5477c-65bfd47b98-l9h9n"]
Jul 28 22:36:45.324236 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:45.324215    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-72a5477c-65bfd47b98-l9h9n"]
Jul 28 22:36:45.338014 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:45.337988    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-72a5477c-8f8d88dfc-6mbhz"]
Jul 28 22:36:45.343573 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:45.343552    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-72a5477c-8f8d88dfc-6mbhz"]
Jul 28 22:36:47.129952 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:47.129921    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c34c245-30af-46c7-a42e-eddec981cd64" path="/var/lib/kubelet/pods/5c34c245-30af-46c7-a42e-eddec981cd64/volumes"
Jul 28 22:36:47.130404 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:47.130270    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="949e3a69-a086-4c74-bd35-12ee72dcd62f" path="/var/lib/kubelet/pods/949e3a69-a086-4c74-bd35-12ee72dcd62f/volumes"
Jul 28 22:36:48.108304 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:48.108274    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b99011fa-5d57bddfd4-sh8wt"]
Jul 28 22:36:48.108834 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:48.108809    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c34c245-30af-46c7-a42e-eddec981cd64" containerName="maas-api"
Jul 28 22:36:48.108834 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:48.108835    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c34c245-30af-46c7-a42e-eddec981cd64" containerName="maas-api"
Jul 28 22:36:48.108981 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:48.108882    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="949e3a69-a086-4c74-bd35-12ee72dcd62f" containerName="payload-processing"
Jul 28 22:36:48.108981 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:48.108892    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="949e3a69-a086-4c74-bd35-12ee72dcd62f" containerName="payload-processing"
Jul 28 22:36:48.109047 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:48.108981    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="949e3a69-a086-4c74-bd35-12ee72dcd62f" containerName="payload-processing"
Jul 28 22:36:48.109047 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:48.108992    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c34c245-30af-46c7-a42e-eddec981cd64" containerName="maas-api"
Jul 28 22:36:48.113745 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:48.113724    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b99011fa-5d57bddfd4-sh8wt"
Jul 28 22:36:48.116237 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:48.116220    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b99011fa-serving-cert\""
Jul 28 22:36:48.121859 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:48.121836    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b99011fa-5d57bddfd4-sh8wt"]
Jul 28 22:36:48.261211 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:48.261174    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxthc\" (UniqueName: \"kubernetes.io/projected/76a201a9-3ba8-4a42-9083-07c5b1ebc316-kube-api-access-zxthc\") pod \"maas-api-e2e-ait-b99011fa-5d57bddfd4-sh8wt\" (UID: \"76a201a9-3ba8-4a42-9083-07c5b1ebc316\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b99011fa-5d57bddfd4-sh8wt"
Jul 28 22:36:48.261636 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:48.261250    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76a201a9-3ba8-4a42-9083-07c5b1ebc316-openshift-service-ca\") pod \"maas-api-e2e-ait-b99011fa-5d57bddfd4-sh8wt\" (UID: \"76a201a9-3ba8-4a42-9083-07c5b1ebc316\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b99011fa-5d57bddfd4-sh8wt"
Jul 28 22:36:48.261636 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:48.261283    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76a201a9-3ba8-4a42-9083-07c5b1ebc316-maas-api-tls\") pod \"maas-api-e2e-ait-b99011fa-5d57bddfd4-sh8wt\" (UID: \"76a201a9-3ba8-4a42-9083-07c5b1ebc316\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b99011fa-5d57bddfd4-sh8wt"
Jul 28 22:36:48.361933 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:48.361843    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76a201a9-3ba8-4a42-9083-07c5b1ebc316-openshift-service-ca\") pod \"maas-api-e2e-ait-b99011fa-5d57bddfd4-sh8wt\" (UID: \"76a201a9-3ba8-4a42-9083-07c5b1ebc316\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b99011fa-5d57bddfd4-sh8wt"
Jul 28 22:36:48.361933 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:48.361880    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76a201a9-3ba8-4a42-9083-07c5b1ebc316-maas-api-tls\") pod \"maas-api-e2e-ait-b99011fa-5d57bddfd4-sh8wt\" (UID: \"76a201a9-3ba8-4a42-9083-07c5b1ebc316\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b99011fa-5d57bddfd4-sh8wt"
Jul 28 22:36:48.361933 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:48.361929    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxthc\" (UniqueName: \"kubernetes.io/projected/76a201a9-3ba8-4a42-9083-07c5b1ebc316-kube-api-access-zxthc\") pod \"maas-api-e2e-ait-b99011fa-5d57bddfd4-sh8wt\" (UID: \"76a201a9-3ba8-4a42-9083-07c5b1ebc316\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b99011fa-5d57bddfd4-sh8wt"
Jul 28 22:36:48.362217 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:36:48.362050    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-b99011fa-serving-cert: secret "maas-api-e2e-ait-b99011fa-serving-cert" not found
Jul 28 22:36:48.362217 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:36:48.362117    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76a201a9-3ba8-4a42-9083-07c5b1ebc316-maas-api-tls podName:76a201a9-3ba8-4a42-9083-07c5b1ebc316 nodeName:}" failed. No retries permitted until 2026-07-28 22:36:48.862096872 +0000 UTC m=+1906.242234495 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/76a201a9-3ba8-4a42-9083-07c5b1ebc316-maas-api-tls") pod "maas-api-e2e-ait-b99011fa-5d57bddfd4-sh8wt" (UID: "76a201a9-3ba8-4a42-9083-07c5b1ebc316") : secret "maas-api-e2e-ait-b99011fa-serving-cert" not found
Jul 28 22:36:48.362699 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:48.362673    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76a201a9-3ba8-4a42-9083-07c5b1ebc316-openshift-service-ca\") pod \"maas-api-e2e-ait-b99011fa-5d57bddfd4-sh8wt\" (UID: \"76a201a9-3ba8-4a42-9083-07c5b1ebc316\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b99011fa-5d57bddfd4-sh8wt"
Jul 28 22:36:48.373267 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:48.373240    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxthc\" (UniqueName: \"kubernetes.io/projected/76a201a9-3ba8-4a42-9083-07c5b1ebc316-kube-api-access-zxthc\") pod \"maas-api-e2e-ait-b99011fa-5d57bddfd4-sh8wt\" (UID: \"76a201a9-3ba8-4a42-9083-07c5b1ebc316\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b99011fa-5d57bddfd4-sh8wt"
Jul 28 22:36:48.867408 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:48.867347    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76a201a9-3ba8-4a42-9083-07c5b1ebc316-maas-api-tls\") pod \"maas-api-e2e-ait-b99011fa-5d57bddfd4-sh8wt\" (UID: \"76a201a9-3ba8-4a42-9083-07c5b1ebc316\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b99011fa-5d57bddfd4-sh8wt"
Jul 28 22:36:48.869869 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:48.869847    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76a201a9-3ba8-4a42-9083-07c5b1ebc316-maas-api-tls\") pod \"maas-api-e2e-ait-b99011fa-5d57bddfd4-sh8wt\" (UID: \"76a201a9-3ba8-4a42-9083-07c5b1ebc316\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b99011fa-5d57bddfd4-sh8wt"
Jul 28 22:36:48.901201 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:48.901165    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b99011fa-6697b66fd8-479qg"]
Jul 28 22:36:48.904728 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:48.904707    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b99011fa-6697b66fd8-479qg"
Jul 28 22:36:48.907271 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:48.907251    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b99011fa\""
Jul 28 22:36:48.919891 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:48.919865    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b99011fa-6697b66fd8-479qg"]
Jul 28 22:36:48.939332 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:48.939307    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b99011fa-6797d66dcd-rfk9j"]
Jul 28 22:36:48.942839 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:48.942825    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b99011fa-6797d66dcd-rfk9j"
Jul 28 22:36:48.945296 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:48.945275    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b99011fa-dockercfg-ncdpr\""
Jul 28 22:36:48.952939 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:48.952912    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b99011fa-6797d66dcd-rfk9j"]
Jul 28 22:36:48.968301 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:48.968275    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1a4487f-245f-4863-8486-0da6e80041b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b99011fa-6697b66fd8-479qg\" (UID: \"f1a4487f-245f-4863-8486-0da6e80041b7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b99011fa-6697b66fd8-479qg"
Jul 28 22:36:48.968445 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:48.968376    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7462\" (UniqueName: \"kubernetes.io/projected/f1a4487f-245f-4863-8486-0da6e80041b7-kube-api-access-f7462\") pod \"payload-pre-processing-e2e-ait-b99011fa-6697b66fd8-479qg\" (UID: \"f1a4487f-245f-4863-8486-0da6e80041b7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b99011fa-6697b66fd8-479qg"
Jul 28 22:36:49.024944 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:49.024915    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b99011fa-5d57bddfd4-sh8wt"
Jul 28 22:36:49.070146 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:49.069536    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1a4487f-245f-4863-8486-0da6e80041b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b99011fa-6697b66fd8-479qg\" (UID: \"f1a4487f-245f-4863-8486-0da6e80041b7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b99011fa-6697b66fd8-479qg"
Jul 28 22:36:49.070146 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:49.069663    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7462\" (UniqueName: \"kubernetes.io/projected/f1a4487f-245f-4863-8486-0da6e80041b7-kube-api-access-f7462\") pod \"payload-pre-processing-e2e-ait-b99011fa-6697b66fd8-479qg\" (UID: \"f1a4487f-245f-4863-8486-0da6e80041b7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b99011fa-6697b66fd8-479qg"
Jul 28 22:36:49.070146 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:49.069712    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3336c82-ce6d-4aca-9cad-c2e706952ed4-plugins-config-volume\") pod \"payload-processing-e2e-ait-b99011fa-6797d66dcd-rfk9j\" (UID: \"b3336c82-ce6d-4aca-9cad-c2e706952ed4\") " pod="openshift-ingress/payload-processing-e2e-ait-b99011fa-6797d66dcd-rfk9j"
Jul 28 22:36:49.070146 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:49.069760    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g94v4\" (UniqueName: \"kubernetes.io/projected/b3336c82-ce6d-4aca-9cad-c2e706952ed4-kube-api-access-g94v4\") pod \"payload-processing-e2e-ait-b99011fa-6797d66dcd-rfk9j\" (UID: \"b3336c82-ce6d-4aca-9cad-c2e706952ed4\") " pod="openshift-ingress/payload-processing-e2e-ait-b99011fa-6797d66dcd-rfk9j"
Jul 28 22:36:49.070515 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:49.070335    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1a4487f-245f-4863-8486-0da6e80041b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b99011fa-6697b66fd8-479qg\" (UID: \"f1a4487f-245f-4863-8486-0da6e80041b7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b99011fa-6697b66fd8-479qg"
Jul 28 22:36:49.078095 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:49.078070    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7462\" (UniqueName: \"kubernetes.io/projected/f1a4487f-245f-4863-8486-0da6e80041b7-kube-api-access-f7462\") pod \"payload-pre-processing-e2e-ait-b99011fa-6697b66fd8-479qg\" (UID: \"f1a4487f-245f-4863-8486-0da6e80041b7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b99011fa-6697b66fd8-479qg"
Jul 28 22:36:49.170917 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:49.170824    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3336c82-ce6d-4aca-9cad-c2e706952ed4-plugins-config-volume\") pod \"payload-processing-e2e-ait-b99011fa-6797d66dcd-rfk9j\" (UID: \"b3336c82-ce6d-4aca-9cad-c2e706952ed4\") " pod="openshift-ingress/payload-processing-e2e-ait-b99011fa-6797d66dcd-rfk9j"
Jul 28 22:36:49.170917 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:49.170876    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g94v4\" (UniqueName: \"kubernetes.io/projected/b3336c82-ce6d-4aca-9cad-c2e706952ed4-kube-api-access-g94v4\") pod \"payload-processing-e2e-ait-b99011fa-6797d66dcd-rfk9j\" (UID: \"b3336c82-ce6d-4aca-9cad-c2e706952ed4\") " pod="openshift-ingress/payload-processing-e2e-ait-b99011fa-6797d66dcd-rfk9j"
Jul 28 22:36:49.171639 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:49.171617    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3336c82-ce6d-4aca-9cad-c2e706952ed4-plugins-config-volume\") pod \"payload-processing-e2e-ait-b99011fa-6797d66dcd-rfk9j\" (UID: \"b3336c82-ce6d-4aca-9cad-c2e706952ed4\") " pod="openshift-ingress/payload-processing-e2e-ait-b99011fa-6797d66dcd-rfk9j"
Jul 28 22:36:49.178225 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:49.178205    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g94v4\" (UniqueName: \"kubernetes.io/projected/b3336c82-ce6d-4aca-9cad-c2e706952ed4-kube-api-access-g94v4\") pod \"payload-processing-e2e-ait-b99011fa-6797d66dcd-rfk9j\" (UID: \"b3336c82-ce6d-4aca-9cad-c2e706952ed4\") " pod="openshift-ingress/payload-processing-e2e-ait-b99011fa-6797d66dcd-rfk9j"
Jul 28 22:36:49.215094 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:49.215071    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b99011fa-6697b66fd8-479qg"
Jul 28 22:36:49.254470 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:49.254439    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b99011fa-6797d66dcd-rfk9j"
Jul 28 22:36:49.363068 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:49.363045    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b99011fa-6697b66fd8-479qg"]
Jul 28 22:36:49.366631 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:36:49.366599    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1a4487f_245f_4863_8486_0da6e80041b7.slice/crio-d3190b61abaf215d801ceb662bcefd46fa278c3e449c103d49f022e26c8f728c WatchSource:0}: Error finding container d3190b61abaf215d801ceb662bcefd46fa278c3e449c103d49f022e26c8f728c: Status 404 returned error can't find the container with id d3190b61abaf215d801ceb662bcefd46fa278c3e449c103d49f022e26c8f728c
Jul 28 22:36:49.376475 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:49.376112    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b99011fa-5d57bddfd4-sh8wt"]
Jul 28 22:36:49.379576 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:36:49.378236    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76a201a9_3ba8_4a42_9083_07c5b1ebc316.slice/crio-612bc7e73fa09d1e59729a60b9a9091a6b572a80b70c97f5a9d8483453a73468 WatchSource:0}: Error finding container 612bc7e73fa09d1e59729a60b9a9091a6b572a80b70c97f5a9d8483453a73468: Status 404 returned error can't find the container with id 612bc7e73fa09d1e59729a60b9a9091a6b572a80b70c97f5a9d8483453a73468
Jul 28 22:36:49.426782 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:49.426648    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b99011fa-6797d66dcd-rfk9j"]
Jul 28 22:36:49.429114 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:36:49.429084    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3336c82_ce6d_4aca_9cad_c2e706952ed4.slice/crio-45b5a47f41770858fca9d9d75dfc917eb57e93c551d4ac90412a66dc100c584c WatchSource:0}: Error finding container 45b5a47f41770858fca9d9d75dfc917eb57e93c551d4ac90412a66dc100c584c: Status 404 returned error can't find the container with id 45b5a47f41770858fca9d9d75dfc917eb57e93c551d4ac90412a66dc100c584c
Jul 28 22:36:50.328842 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:50.328806    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b99011fa-6697b66fd8-479qg" event={"ID":"f1a4487f-245f-4863-8486-0da6e80041b7","Type":"ContainerStarted","Data":"d67a21f3bf526af2ad193ffbee85e689a8997310740975584319e0cc85a93cc2"}
Jul 28 22:36:50.329043 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:50.328869    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b99011fa-6697b66fd8-479qg" event={"ID":"f1a4487f-245f-4863-8486-0da6e80041b7","Type":"ContainerStarted","Data":"d3190b61abaf215d801ceb662bcefd46fa278c3e449c103d49f022e26c8f728c"}
Jul 28 22:36:50.329043 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:50.328896    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b99011fa-6697b66fd8-479qg"
Jul 28 22:36:50.330532 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:50.330504    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b99011fa-6797d66dcd-rfk9j" event={"ID":"b3336c82-ce6d-4aca-9cad-c2e706952ed4","Type":"ContainerStarted","Data":"08b6cb3a8655f12b027bdda3d850d1ce47189ce4ee05beac677dd48b9472f1ab"}
Jul 28 22:36:50.330734 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:50.330559    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b99011fa-6797d66dcd-rfk9j"
Jul 28 22:36:50.330734 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:50.330585    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b99011fa-6797d66dcd-rfk9j" event={"ID":"b3336c82-ce6d-4aca-9cad-c2e706952ed4","Type":"ContainerStarted","Data":"45b5a47f41770858fca9d9d75dfc917eb57e93c551d4ac90412a66dc100c584c"}
Jul 28 22:36:50.331995 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:50.331967    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b99011fa-5d57bddfd4-sh8wt" event={"ID":"76a201a9-3ba8-4a42-9083-07c5b1ebc316","Type":"ContainerStarted","Data":"80a7fb71789ad391139722b4b6e81dd44314df93ae0cdd669cf67835aa90f1e7"}
Jul 28 22:36:50.332122 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:50.332007    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b99011fa-5d57bddfd4-sh8wt" event={"ID":"76a201a9-3ba8-4a42-9083-07c5b1ebc316","Type":"ContainerStarted","Data":"612bc7e73fa09d1e59729a60b9a9091a6b572a80b70c97f5a9d8483453a73468"}
Jul 28 22:36:50.332122 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:50.332033    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b99011fa-5d57bddfd4-sh8wt"
Jul 28 22:36:50.349294 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:50.349236    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b99011fa-6697b66fd8-479qg" podStartSLOduration=2.349220576 podStartE2EDuration="2.349220576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:36:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:36:50.348376952 +0000 UTC m=+1907.728514619" watchObservedRunningTime="2026-07-28 22:36:50.349220576 +0000 UTC m=+1907.729358220"
Jul 28 22:36:50.365110 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:50.365070    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b99011fa-5d57bddfd4-sh8wt" podStartSLOduration=1.8857901799999999 podStartE2EDuration="2.365055748s" totalImagesPullingTime="479.265568ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:36:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:36:50.363827665 +0000 UTC m=+1907.743965310" watchObservedRunningTime="2026-07-28 22:36:50.365055748 +0000 UTC m=+1907.745193393"
Jul 28 22:36:50.381700 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:50.381654    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b99011fa-6797d66dcd-rfk9j" podStartSLOduration=2.381644724 podStartE2EDuration="2.381644724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:36:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:36:50.380784284 +0000 UTC m=+1907.760921918" watchObservedRunningTime="2026-07-28 22:36:50.381644724 +0000 UTC m=+1907.761782367"
Jul 28 22:36:56.343176 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:36:56.343141    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b99011fa-5d57bddfd4-sh8wt"
Jul 28 22:37:01.340276 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:01.340240    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b99011fa-6797d66dcd-rfk9j"
Jul 28 22:37:01.340778 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:01.340758    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b99011fa-6697b66fd8-479qg"
Jul 28 22:37:18.816465 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:18.816422    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b99011fa-5d57bddfd4-sh8wt"]
Jul 28 22:37:18.817027 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:18.816692    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b99011fa-5d57bddfd4-sh8wt" podUID="76a201a9-3ba8-4a42-9083-07c5b1ebc316" containerName="maas-api" containerID="cri-o://80a7fb71789ad391139722b4b6e81dd44314df93ae0cdd669cf67835aa90f1e7" gracePeriod=30
Jul 28 22:37:18.888667 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:18.888626    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b99011fa-6797d66dcd-rfk9j"]
Jul 28 22:37:18.888962 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:18.888904    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b99011fa-6797d66dcd-rfk9j" podUID="b3336c82-ce6d-4aca-9cad-c2e706952ed4" containerName="payload-processing" containerID="cri-o://08b6cb3a8655f12b027bdda3d850d1ce47189ce4ee05beac677dd48b9472f1ab" gracePeriod=30
Jul 28 22:37:18.900410 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:18.900382    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b99011fa-6697b66fd8-479qg"]
Jul 28 22:37:18.900840 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:18.900795    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b99011fa-6697b66fd8-479qg" podUID="f1a4487f-245f-4863-8486-0da6e80041b7" containerName="payload-pre-processing" containerID="cri-o://d67a21f3bf526af2ad193ffbee85e689a8997310740975584319e0cc85a93cc2" gracePeriod=30
Jul 28 22:37:19.170262 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.170236    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b99011fa-5d57bddfd4-sh8wt"
Jul 28 22:37:19.181409 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.181387    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b99011fa-6697b66fd8-479qg"
Jul 28 22:37:19.194092 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.194073    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b99011fa-6797d66dcd-rfk9j"
Jul 28 22:37:19.242366 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.242325    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3336c82-ce6d-4aca-9cad-c2e706952ed4-plugins-config-volume\") pod \"b3336c82-ce6d-4aca-9cad-c2e706952ed4\" (UID: \"b3336c82-ce6d-4aca-9cad-c2e706952ed4\") "
Jul 28 22:37:19.242653 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.242429    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1a4487f-245f-4863-8486-0da6e80041b7-plugins-config-volume\") pod \"f1a4487f-245f-4863-8486-0da6e80041b7\" (UID: \"f1a4487f-245f-4863-8486-0da6e80041b7\") "
Jul 28 22:37:19.242653 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.242479    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxthc\" (UniqueName: \"kubernetes.io/projected/76a201a9-3ba8-4a42-9083-07c5b1ebc316-kube-api-access-zxthc\") pod \"76a201a9-3ba8-4a42-9083-07c5b1ebc316\" (UID: \"76a201a9-3ba8-4a42-9083-07c5b1ebc316\") "
Jul 28 22:37:19.242653 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.242509    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76a201a9-3ba8-4a42-9083-07c5b1ebc316-maas-api-tls\") pod \"76a201a9-3ba8-4a42-9083-07c5b1ebc316\" (UID: \"76a201a9-3ba8-4a42-9083-07c5b1ebc316\") "
Jul 28 22:37:19.242653 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.242536    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7462\" (UniqueName: \"kubernetes.io/projected/f1a4487f-245f-4863-8486-0da6e80041b7-kube-api-access-f7462\") pod \"f1a4487f-245f-4863-8486-0da6e80041b7\" (UID: \"f1a4487f-245f-4863-8486-0da6e80041b7\") "
Jul 28 22:37:19.242653 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.242567    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76a201a9-3ba8-4a42-9083-07c5b1ebc316-openshift-service-ca\") pod \"76a201a9-3ba8-4a42-9083-07c5b1ebc316\" (UID: \"76a201a9-3ba8-4a42-9083-07c5b1ebc316\") "
Jul 28 22:37:19.242653 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.242592    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g94v4\" (UniqueName: \"kubernetes.io/projected/b3336c82-ce6d-4aca-9cad-c2e706952ed4-kube-api-access-g94v4\") pod \"b3336c82-ce6d-4aca-9cad-c2e706952ed4\" (UID: \"b3336c82-ce6d-4aca-9cad-c2e706952ed4\") "
Jul 28 22:37:19.243659 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.243261    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3336c82-ce6d-4aca-9cad-c2e706952ed4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b3336c82-ce6d-4aca-9cad-c2e706952ed4" (UID: "b3336c82-ce6d-4aca-9cad-c2e706952ed4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:37:19.243875 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.243833    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76a201a9-3ba8-4a42-9083-07c5b1ebc316-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "76a201a9-3ba8-4a42-9083-07c5b1ebc316" (UID: "76a201a9-3ba8-4a42-9083-07c5b1ebc316"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:37:19.243963 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.243938    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1a4487f-245f-4863-8486-0da6e80041b7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f1a4487f-245f-4863-8486-0da6e80041b7" (UID: "f1a4487f-245f-4863-8486-0da6e80041b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:37:19.246064 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.246033    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3336c82-ce6d-4aca-9cad-c2e706952ed4-kube-api-access-g94v4" (OuterVolumeSpecName: "kube-api-access-g94v4") pod "b3336c82-ce6d-4aca-9cad-c2e706952ed4" (UID: "b3336c82-ce6d-4aca-9cad-c2e706952ed4"). InnerVolumeSpecName "kube-api-access-g94v4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:37:19.246172 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.246067    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76a201a9-3ba8-4a42-9083-07c5b1ebc316-kube-api-access-zxthc" (OuterVolumeSpecName: "kube-api-access-zxthc") pod "76a201a9-3ba8-4a42-9083-07c5b1ebc316" (UID: "76a201a9-3ba8-4a42-9083-07c5b1ebc316"). InnerVolumeSpecName "kube-api-access-zxthc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:37:19.246283 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.246258    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1a4487f-245f-4863-8486-0da6e80041b7-kube-api-access-f7462" (OuterVolumeSpecName: "kube-api-access-f7462") pod "f1a4487f-245f-4863-8486-0da6e80041b7" (UID: "f1a4487f-245f-4863-8486-0da6e80041b7"). InnerVolumeSpecName "kube-api-access-f7462". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:37:19.246408 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.246388    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76a201a9-3ba8-4a42-9083-07c5b1ebc316-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "76a201a9-3ba8-4a42-9083-07c5b1ebc316" (UID: "76a201a9-3ba8-4a42-9083-07c5b1ebc316"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:37:19.343406 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.343298    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76a201a9-3ba8-4a42-9083-07c5b1ebc316-openshift-service-ca\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:37:19.343406 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.343327    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g94v4\" (UniqueName: \"kubernetes.io/projected/b3336c82-ce6d-4aca-9cad-c2e706952ed4-kube-api-access-g94v4\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:37:19.343406 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.343340    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3336c82-ce6d-4aca-9cad-c2e706952ed4-plugins-config-volume\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:37:19.343406 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.343376    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1a4487f-245f-4863-8486-0da6e80041b7-plugins-config-volume\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:37:19.343406 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.343390    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxthc\" (UniqueName: \"kubernetes.io/projected/76a201a9-3ba8-4a42-9083-07c5b1ebc316-kube-api-access-zxthc\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:37:19.343406 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.343403    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76a201a9-3ba8-4a42-9083-07c5b1ebc316-maas-api-tls\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:37:19.343701 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.343415    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7462\" (UniqueName: \"kubernetes.io/projected/f1a4487f-245f-4863-8486-0da6e80041b7-kube-api-access-f7462\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:37:19.476950 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.476916    2563 generic.go:356] "Generic (PLEG): container finished" podID="f1a4487f-245f-4863-8486-0da6e80041b7" containerID="d67a21f3bf526af2ad193ffbee85e689a8997310740975584319e0cc85a93cc2" exitCode=0
Jul 28 22:37:19.477157 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.476964    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b99011fa-6697b66fd8-479qg"
Jul 28 22:37:19.477157 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.477022    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b99011fa-6697b66fd8-479qg" event={"ID":"f1a4487f-245f-4863-8486-0da6e80041b7","Type":"ContainerDied","Data":"d67a21f3bf526af2ad193ffbee85e689a8997310740975584319e0cc85a93cc2"}
Jul 28 22:37:19.477157 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.477077    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b99011fa-6697b66fd8-479qg" event={"ID":"f1a4487f-245f-4863-8486-0da6e80041b7","Type":"ContainerDied","Data":"d3190b61abaf215d801ceb662bcefd46fa278c3e449c103d49f022e26c8f728c"}
Jul 28 22:37:19.477157 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.477107    2563 scope.go:117] "RemoveContainer" containerID="d67a21f3bf526af2ad193ffbee85e689a8997310740975584319e0cc85a93cc2"
Jul 28 22:37:19.480671 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.478721    2563 generic.go:356] "Generic (PLEG): container finished" podID="b3336c82-ce6d-4aca-9cad-c2e706952ed4" containerID="08b6cb3a8655f12b027bdda3d850d1ce47189ce4ee05beac677dd48b9472f1ab" exitCode=0
Jul 28 22:37:19.480671 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.478806    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b99011fa-6797d66dcd-rfk9j" event={"ID":"b3336c82-ce6d-4aca-9cad-c2e706952ed4","Type":"ContainerDied","Data":"08b6cb3a8655f12b027bdda3d850d1ce47189ce4ee05beac677dd48b9472f1ab"}
Jul 28 22:37:19.480671 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.478843    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b99011fa-6797d66dcd-rfk9j" event={"ID":"b3336c82-ce6d-4aca-9cad-c2e706952ed4","Type":"ContainerDied","Data":"45b5a47f41770858fca9d9d75dfc917eb57e93c551d4ac90412a66dc100c584c"}
Jul 28 22:37:19.480671 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.478901    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b99011fa-6797d66dcd-rfk9j"
Jul 28 22:37:19.482103 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.482077    2563 generic.go:356] "Generic (PLEG): container finished" podID="76a201a9-3ba8-4a42-9083-07c5b1ebc316" containerID="80a7fb71789ad391139722b4b6e81dd44314df93ae0cdd669cf67835aa90f1e7" exitCode=0
Jul 28 22:37:19.482232 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.482130    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b99011fa-5d57bddfd4-sh8wt"
Jul 28 22:37:19.482232 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.482155    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b99011fa-5d57bddfd4-sh8wt" event={"ID":"76a201a9-3ba8-4a42-9083-07c5b1ebc316","Type":"ContainerDied","Data":"80a7fb71789ad391139722b4b6e81dd44314df93ae0cdd669cf67835aa90f1e7"}
Jul 28 22:37:19.482232 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.482197    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b99011fa-5d57bddfd4-sh8wt" event={"ID":"76a201a9-3ba8-4a42-9083-07c5b1ebc316","Type":"ContainerDied","Data":"612bc7e73fa09d1e59729a60b9a9091a6b572a80b70c97f5a9d8483453a73468"}
Jul 28 22:37:19.488942 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.488920    2563 scope.go:117] "RemoveContainer" containerID="d67a21f3bf526af2ad193ffbee85e689a8997310740975584319e0cc85a93cc2"
Jul 28 22:37:19.489228 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:37:19.489202    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d67a21f3bf526af2ad193ffbee85e689a8997310740975584319e0cc85a93cc2\": container with ID starting with d67a21f3bf526af2ad193ffbee85e689a8997310740975584319e0cc85a93cc2 not found: ID does not exist" containerID="d67a21f3bf526af2ad193ffbee85e689a8997310740975584319e0cc85a93cc2"
Jul 28 22:37:19.489339 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.489238    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67a21f3bf526af2ad193ffbee85e689a8997310740975584319e0cc85a93cc2"} err="failed to get container status \"d67a21f3bf526af2ad193ffbee85e689a8997310740975584319e0cc85a93cc2\": rpc error: code = NotFound desc = could not find container \"d67a21f3bf526af2ad193ffbee85e689a8997310740975584319e0cc85a93cc2\": container with ID starting with d67a21f3bf526af2ad193ffbee85e689a8997310740975584319e0cc85a93cc2 not found: ID does not exist"
Jul 28 22:37:19.489339 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.489267    2563 scope.go:117] "RemoveContainer" containerID="08b6cb3a8655f12b027bdda3d850d1ce47189ce4ee05beac677dd48b9472f1ab"
Jul 28 22:37:19.497704 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.497687    2563 scope.go:117] "RemoveContainer" containerID="08b6cb3a8655f12b027bdda3d850d1ce47189ce4ee05beac677dd48b9472f1ab"
Jul 28 22:37:19.497929 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:37:19.497913    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08b6cb3a8655f12b027bdda3d850d1ce47189ce4ee05beac677dd48b9472f1ab\": container with ID starting with 08b6cb3a8655f12b027bdda3d850d1ce47189ce4ee05beac677dd48b9472f1ab not found: ID does not exist" containerID="08b6cb3a8655f12b027bdda3d850d1ce47189ce4ee05beac677dd48b9472f1ab"
Jul 28 22:37:19.497981 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.497935    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08b6cb3a8655f12b027bdda3d850d1ce47189ce4ee05beac677dd48b9472f1ab"} err="failed to get container status \"08b6cb3a8655f12b027bdda3d850d1ce47189ce4ee05beac677dd48b9472f1ab\": rpc error: code = NotFound desc = could not find container \"08b6cb3a8655f12b027bdda3d850d1ce47189ce4ee05beac677dd48b9472f1ab\": container with ID starting with 08b6cb3a8655f12b027bdda3d850d1ce47189ce4ee05beac677dd48b9472f1ab not found: ID does not exist"
Jul 28 22:37:19.497981 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.497952    2563 scope.go:117] "RemoveContainer" containerID="80a7fb71789ad391139722b4b6e81dd44314df93ae0cdd669cf67835aa90f1e7"
Jul 28 22:37:19.504954 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.504938    2563 scope.go:117] "RemoveContainer" containerID="80a7fb71789ad391139722b4b6e81dd44314df93ae0cdd669cf67835aa90f1e7"
Jul 28 22:37:19.505195 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:37:19.505177    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80a7fb71789ad391139722b4b6e81dd44314df93ae0cdd669cf67835aa90f1e7\": container with ID starting with 80a7fb71789ad391139722b4b6e81dd44314df93ae0cdd669cf67835aa90f1e7 not found: ID does not exist" containerID="80a7fb71789ad391139722b4b6e81dd44314df93ae0cdd669cf67835aa90f1e7"
Jul 28 22:37:19.505233 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.505204    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80a7fb71789ad391139722b4b6e81dd44314df93ae0cdd669cf67835aa90f1e7"} err="failed to get container status \"80a7fb71789ad391139722b4b6e81dd44314df93ae0cdd669cf67835aa90f1e7\": rpc error: code = NotFound desc = could not find container \"80a7fb71789ad391139722b4b6e81dd44314df93ae0cdd669cf67835aa90f1e7\": container with ID starting with 80a7fb71789ad391139722b4b6e81dd44314df93ae0cdd669cf67835aa90f1e7 not found: ID does not exist"
Jul 28 22:37:19.511071 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.511049    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b99011fa-6697b66fd8-479qg"]
Jul 28 22:37:19.525004 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.524983    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b99011fa-6697b66fd8-479qg"]
Jul 28 22:37:19.562741 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.562716    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b99011fa-5d57bddfd4-sh8wt"]
Jul 28 22:37:19.567896 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.567873    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b99011fa-5d57bddfd4-sh8wt"]
Jul 28 22:37:19.577997 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.577976    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b99011fa-6797d66dcd-rfk9j"]
Jul 28 22:37:19.581878 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:19.581859    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b99011fa-6797d66dcd-rfk9j"]
Jul 28 22:37:21.130568 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:21.130537    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76a201a9-3ba8-4a42-9083-07c5b1ebc316" path="/var/lib/kubelet/pods/76a201a9-3ba8-4a42-9083-07c5b1ebc316/volumes"
Jul 28 22:37:21.130944 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:21.130890    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3336c82-ce6d-4aca-9cad-c2e706952ed4" path="/var/lib/kubelet/pods/b3336c82-ce6d-4aca-9cad-c2e706952ed4/volumes"
Jul 28 22:37:21.131198 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:21.131185    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1a4487f-245f-4863-8486-0da6e80041b7" path="/var/lib/kubelet/pods/f1a4487f-245f-4863-8486-0da6e80041b7/volumes"
Jul 28 22:37:23.228186 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:23.228141    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3db5e9ba-68b64685c-fr4kj"]
Jul 28 22:37:23.228758 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:23.228736    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1a4487f-245f-4863-8486-0da6e80041b7" containerName="payload-pre-processing"
Jul 28 22:37:23.228813 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:23.228763    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1a4487f-245f-4863-8486-0da6e80041b7" containerName="payload-pre-processing"
Jul 28 22:37:23.228813 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:23.228778    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3336c82-ce6d-4aca-9cad-c2e706952ed4" containerName="payload-processing"
Jul 28 22:37:23.228813 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:23.228788    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3336c82-ce6d-4aca-9cad-c2e706952ed4" containerName="payload-processing"
Jul 28 22:37:23.228813 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:23.228799    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76a201a9-3ba8-4a42-9083-07c5b1ebc316" containerName="maas-api"
Jul 28 22:37:23.228813 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:23.228809    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="76a201a9-3ba8-4a42-9083-07c5b1ebc316" containerName="maas-api"
Jul 28 22:37:23.229003 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:23.228907    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="76a201a9-3ba8-4a42-9083-07c5b1ebc316" containerName="maas-api"
Jul 28 22:37:23.229003 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:23.228921    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1a4487f-245f-4863-8486-0da6e80041b7" containerName="payload-pre-processing"
Jul 28 22:37:23.229003 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:23.228934    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3336c82-ce6d-4aca-9cad-c2e706952ed4" containerName="payload-processing"
Jul 28 22:37:23.233887 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:23.233866    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3db5e9ba-68b64685c-fr4kj"
Jul 28 22:37:23.236617 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:23.236595    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-3db5e9ba-serving-cert\""
Jul 28 22:37:23.243046 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:23.243024    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3db5e9ba-68b64685c-fr4kj"]
Jul 28 22:37:23.279209 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:23.279185    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a3eb38f-5e6c-4802-bb0a-6b477fe716fb-maas-api-tls\") pod \"maas-api-e2e-derive-3db5e9ba-68b64685c-fr4kj\" (UID: \"4a3eb38f-5e6c-4802-bb0a-6b477fe716fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3db5e9ba-68b64685c-fr4kj"
Jul 28 22:37:23.279338 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:23.279212    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a3eb38f-5e6c-4802-bb0a-6b477fe716fb-openshift-service-ca\") pod \"maas-api-e2e-derive-3db5e9ba-68b64685c-fr4kj\" (UID: \"4a3eb38f-5e6c-4802-bb0a-6b477fe716fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3db5e9ba-68b64685c-fr4kj"
Jul 28 22:37:23.279338 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:23.279305    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df9n5\" (UniqueName: \"kubernetes.io/projected/4a3eb38f-5e6c-4802-bb0a-6b477fe716fb-kube-api-access-df9n5\") pod \"maas-api-e2e-derive-3db5e9ba-68b64685c-fr4kj\" (UID: \"4a3eb38f-5e6c-4802-bb0a-6b477fe716fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3db5e9ba-68b64685c-fr4kj"
Jul 28 22:37:23.380098 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:23.380065    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a3eb38f-5e6c-4802-bb0a-6b477fe716fb-maas-api-tls\") pod \"maas-api-e2e-derive-3db5e9ba-68b64685c-fr4kj\" (UID: \"4a3eb38f-5e6c-4802-bb0a-6b477fe716fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3db5e9ba-68b64685c-fr4kj"
Jul 28 22:37:23.380297 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:23.380102    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a3eb38f-5e6c-4802-bb0a-6b477fe716fb-openshift-service-ca\") pod \"maas-api-e2e-derive-3db5e9ba-68b64685c-fr4kj\" (UID: \"4a3eb38f-5e6c-4802-bb0a-6b477fe716fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3db5e9ba-68b64685c-fr4kj"
Jul 28 22:37:23.380297 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:23.380189    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df9n5\" (UniqueName: \"kubernetes.io/projected/4a3eb38f-5e6c-4802-bb0a-6b477fe716fb-kube-api-access-df9n5\") pod \"maas-api-e2e-derive-3db5e9ba-68b64685c-fr4kj\" (UID: \"4a3eb38f-5e6c-4802-bb0a-6b477fe716fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3db5e9ba-68b64685c-fr4kj"
Jul 28 22:37:23.380297 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:37:23.380217    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-3db5e9ba-serving-cert: secret "maas-api-e2e-derive-3db5e9ba-serving-cert" not found
Jul 28 22:37:23.380474 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:37:23.380304    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a3eb38f-5e6c-4802-bb0a-6b477fe716fb-maas-api-tls podName:4a3eb38f-5e6c-4802-bb0a-6b477fe716fb nodeName:}" failed. No retries permitted until 2026-07-28 22:37:23.880283183 +0000 UTC m=+1941.260420819 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4a3eb38f-5e6c-4802-bb0a-6b477fe716fb-maas-api-tls") pod "maas-api-e2e-derive-3db5e9ba-68b64685c-fr4kj" (UID: "4a3eb38f-5e6c-4802-bb0a-6b477fe716fb") : secret "maas-api-e2e-derive-3db5e9ba-serving-cert" not found
Jul 28 22:37:23.380832 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:23.380809    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a3eb38f-5e6c-4802-bb0a-6b477fe716fb-openshift-service-ca\") pod \"maas-api-e2e-derive-3db5e9ba-68b64685c-fr4kj\" (UID: \"4a3eb38f-5e6c-4802-bb0a-6b477fe716fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3db5e9ba-68b64685c-fr4kj"
Jul 28 22:37:23.390710 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:23.390687    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-df9n5\" (UniqueName: \"kubernetes.io/projected/4a3eb38f-5e6c-4802-bb0a-6b477fe716fb-kube-api-access-df9n5\") pod \"maas-api-e2e-derive-3db5e9ba-68b64685c-fr4kj\" (UID: \"4a3eb38f-5e6c-4802-bb0a-6b477fe716fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3db5e9ba-68b64685c-fr4kj"
Jul 28 22:37:23.824288 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:23.824250    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3db5e9ba-88bf678b4-f4m6w"]
Jul 28 22:37:23.827910 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:23.827888    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3db5e9ba-88bf678b4-f4m6w"
Jul 28 22:37:23.830553 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:23.830529    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-3db5e9ba\""
Jul 28 22:37:23.830662 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:23.830591    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-3db5e9ba-dockercfg-ngqdt\""
Jul 28 22:37:23.840884 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:23.840857    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3db5e9ba-88bf678b4-f4m6w"]
Jul 28 22:37:23.858983 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:23.858955    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3db5e9ba-7c94dc4c6b-w25zl"]
Jul 28 22:37:23.862497 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:23.862479    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3db5e9ba-7c94dc4c6b-w25zl"
Jul 28 22:37:23.872508 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:23.872488    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3db5e9ba-7c94dc4c6b-w25zl"]
Jul 28 22:37:23.885851 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:23.885830    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c89qs\" (UniqueName: \"kubernetes.io/projected/25b9e0e7-f8f0-472b-bc0e-3fc8bfddad36-kube-api-access-c89qs\") pod \"payload-pre-processing-e2e-derive-3db5e9ba-88bf678b4-f4m6w\" (UID: \"25b9e0e7-f8f0-472b-bc0e-3fc8bfddad36\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3db5e9ba-88bf678b4-f4m6w"
Jul 28 22:37:23.885946 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:23.885865    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a3eb38f-5e6c-4802-bb0a-6b477fe716fb-maas-api-tls\") pod \"maas-api-e2e-derive-3db5e9ba-68b64685c-fr4kj\" (UID: \"4a3eb38f-5e6c-4802-bb0a-6b477fe716fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3db5e9ba-68b64685c-fr4kj"
Jul 28 22:37:23.886025 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:23.885999    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25b9e0e7-f8f0-472b-bc0e-3fc8bfddad36-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3db5e9ba-88bf678b4-f4m6w\" (UID: \"25b9e0e7-f8f0-472b-bc0e-3fc8bfddad36\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3db5e9ba-88bf678b4-f4m6w"
Jul 28 22:37:23.888549 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:23.888526    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a3eb38f-5e6c-4802-bb0a-6b477fe716fb-maas-api-tls\") pod \"maas-api-e2e-derive-3db5e9ba-68b64685c-fr4kj\" (UID: \"4a3eb38f-5e6c-4802-bb0a-6b477fe716fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3db5e9ba-68b64685c-fr4kj"
Jul 28 22:37:23.986895 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:23.986859    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwt7j\" (UniqueName: \"kubernetes.io/projected/80c31930-49dd-4e1d-93eb-656693ed0aab-kube-api-access-lwt7j\") pod \"payload-processing-e2e-derive-3db5e9ba-7c94dc4c6b-w25zl\" (UID: \"80c31930-49dd-4e1d-93eb-656693ed0aab\") " pod="openshift-ingress/payload-processing-e2e-derive-3db5e9ba-7c94dc4c6b-w25zl"
Jul 28 22:37:23.987044 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:23.986930    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25b9e0e7-f8f0-472b-bc0e-3fc8bfddad36-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3db5e9ba-88bf678b4-f4m6w\" (UID: \"25b9e0e7-f8f0-472b-bc0e-3fc8bfddad36\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3db5e9ba-88bf678b4-f4m6w"
Jul 28 22:37:23.987044 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:23.986972    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c89qs\" (UniqueName: \"kubernetes.io/projected/25b9e0e7-f8f0-472b-bc0e-3fc8bfddad36-kube-api-access-c89qs\") pod \"payload-pre-processing-e2e-derive-3db5e9ba-88bf678b4-f4m6w\" (UID: \"25b9e0e7-f8f0-472b-bc0e-3fc8bfddad36\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3db5e9ba-88bf678b4-f4m6w"
Jul 28 22:37:23.987172 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:23.987124    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80c31930-49dd-4e1d-93eb-656693ed0aab-plugins-config-volume\") pod \"payload-processing-e2e-derive-3db5e9ba-7c94dc4c6b-w25zl\" (UID: \"80c31930-49dd-4e1d-93eb-656693ed0aab\") " pod="openshift-ingress/payload-processing-e2e-derive-3db5e9ba-7c94dc4c6b-w25zl"
Jul 28 22:37:23.987712 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:23.987693    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25b9e0e7-f8f0-472b-bc0e-3fc8bfddad36-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3db5e9ba-88bf678b4-f4m6w\" (UID: \"25b9e0e7-f8f0-472b-bc0e-3fc8bfddad36\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3db5e9ba-88bf678b4-f4m6w"
Jul 28 22:37:23.994981 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:23.994960    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c89qs\" (UniqueName: \"kubernetes.io/projected/25b9e0e7-f8f0-472b-bc0e-3fc8bfddad36-kube-api-access-c89qs\") pod \"payload-pre-processing-e2e-derive-3db5e9ba-88bf678b4-f4m6w\" (UID: \"25b9e0e7-f8f0-472b-bc0e-3fc8bfddad36\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3db5e9ba-88bf678b4-f4m6w"
Jul 28 22:37:24.087769 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:24.087683    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80c31930-49dd-4e1d-93eb-656693ed0aab-plugins-config-volume\") pod \"payload-processing-e2e-derive-3db5e9ba-7c94dc4c6b-w25zl\" (UID: \"80c31930-49dd-4e1d-93eb-656693ed0aab\") " pod="openshift-ingress/payload-processing-e2e-derive-3db5e9ba-7c94dc4c6b-w25zl"
Jul 28 22:37:24.087769 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:24.087732    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwt7j\" (UniqueName: \"kubernetes.io/projected/80c31930-49dd-4e1d-93eb-656693ed0aab-kube-api-access-lwt7j\") pod \"payload-processing-e2e-derive-3db5e9ba-7c94dc4c6b-w25zl\" (UID: \"80c31930-49dd-4e1d-93eb-656693ed0aab\") " pod="openshift-ingress/payload-processing-e2e-derive-3db5e9ba-7c94dc4c6b-w25zl"
Jul 28 22:37:24.088407 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:24.088389    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80c31930-49dd-4e1d-93eb-656693ed0aab-plugins-config-volume\") pod \"payload-processing-e2e-derive-3db5e9ba-7c94dc4c6b-w25zl\" (UID: \"80c31930-49dd-4e1d-93eb-656693ed0aab\") " pod="openshift-ingress/payload-processing-e2e-derive-3db5e9ba-7c94dc4c6b-w25zl"
Jul 28 22:37:24.095940 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:24.095912    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwt7j\" (UniqueName: \"kubernetes.io/projected/80c31930-49dd-4e1d-93eb-656693ed0aab-kube-api-access-lwt7j\") pod \"payload-processing-e2e-derive-3db5e9ba-7c94dc4c6b-w25zl\" (UID: \"80c31930-49dd-4e1d-93eb-656693ed0aab\") " pod="openshift-ingress/payload-processing-e2e-derive-3db5e9ba-7c94dc4c6b-w25zl"
Jul 28 22:37:24.139648 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:24.139614    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3db5e9ba-88bf678b4-f4m6w"
Jul 28 22:37:24.144675 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:24.144655    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3db5e9ba-68b64685c-fr4kj"
Jul 28 22:37:24.175286 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:24.175251    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3db5e9ba-7c94dc4c6b-w25zl"
Jul 28 22:37:24.292586 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:24.292544    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3db5e9ba-88bf678b4-f4m6w"]
Jul 28 22:37:24.295004 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:37:24.294974    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25b9e0e7_f8f0_472b_bc0e_3fc8bfddad36.slice/crio-049a907c6a7fdd702a3d73283cd8a44c677efa419c40d85718f5a362762342c9 WatchSource:0}: Error finding container 049a907c6a7fdd702a3d73283cd8a44c677efa419c40d85718f5a362762342c9: Status 404 returned error can't find the container with id 049a907c6a7fdd702a3d73283cd8a44c677efa419c40d85718f5a362762342c9
Jul 28 22:37:24.331645 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:37:24.331612    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a3eb38f_5e6c_4802_bb0a_6b477fe716fb.slice/crio-b5c5913929c51bfd84eb96c92a17f16635fd969e09f5053caa1856b380bc1c8a WatchSource:0}: Error finding container b5c5913929c51bfd84eb96c92a17f16635fd969e09f5053caa1856b380bc1c8a: Status 404 returned error can't find the container with id b5c5913929c51bfd84eb96c92a17f16635fd969e09f5053caa1856b380bc1c8a
Jul 28 22:37:24.331838 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:24.331685    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3db5e9ba-68b64685c-fr4kj"]
Jul 28 22:37:24.357673 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:24.357650    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3db5e9ba-7c94dc4c6b-w25zl"]
Jul 28 22:37:24.359568 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:37:24.359499    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80c31930_49dd_4e1d_93eb_656693ed0aab.slice/crio-b52b08ca9eb9f292b4747d3b588f0b540ca79b4b8711bbc5aa32ac45e79b4ef9 WatchSource:0}: Error finding container b52b08ca9eb9f292b4747d3b588f0b540ca79b4b8711bbc5aa32ac45e79b4ef9: Status 404 returned error can't find the container with id b52b08ca9eb9f292b4747d3b588f0b540ca79b4b8711bbc5aa32ac45e79b4ef9
Jul 28 22:37:24.509736 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:24.509704    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3db5e9ba-68b64685c-fr4kj" event={"ID":"4a3eb38f-5e6c-4802-bb0a-6b477fe716fb","Type":"ContainerStarted","Data":"b5c5913929c51bfd84eb96c92a17f16635fd969e09f5053caa1856b380bc1c8a"}
Jul 28 22:37:24.511123 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:24.511098    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3db5e9ba-7c94dc4c6b-w25zl" event={"ID":"80c31930-49dd-4e1d-93eb-656693ed0aab","Type":"ContainerStarted","Data":"1e7868d640da2e0d095ab231c0e445c964f59a509e31cb18f3702cd19b0314d6"}
Jul 28 22:37:24.511249 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:24.511148    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3db5e9ba-7c94dc4c6b-w25zl" event={"ID":"80c31930-49dd-4e1d-93eb-656693ed0aab","Type":"ContainerStarted","Data":"b52b08ca9eb9f292b4747d3b588f0b540ca79b4b8711bbc5aa32ac45e79b4ef9"}
Jul 28 22:37:24.511249 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:24.511221    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-3db5e9ba-7c94dc4c6b-w25zl"
Jul 28 22:37:24.512458 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:24.512434    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3db5e9ba-88bf678b4-f4m6w" event={"ID":"25b9e0e7-f8f0-472b-bc0e-3fc8bfddad36","Type":"ContainerStarted","Data":"8d09c03ec6a12964cf0e8008fa7081a19dd5e4aa9557b05196544bce58f0c017"}
Jul 28 22:37:24.512548 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:24.512477    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-3db5e9ba-88bf678b4-f4m6w"
Jul 28 22:37:24.512548 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:24.512492    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3db5e9ba-88bf678b4-f4m6w" event={"ID":"25b9e0e7-f8f0-472b-bc0e-3fc8bfddad36","Type":"ContainerStarted","Data":"049a907c6a7fdd702a3d73283cd8a44c677efa419c40d85718f5a362762342c9"}
Jul 28 22:37:24.527403 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:24.527336    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-3db5e9ba-7c94dc4c6b-w25zl" podStartSLOduration=1.5273187940000001 podStartE2EDuration="1.527318794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:37:24.526757719 +0000 UTC m=+1941.906895438" watchObservedRunningTime="2026-07-28 22:37:24.527318794 +0000 UTC m=+1941.907456439"
Jul 28 22:37:24.541799 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:24.541750    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-3db5e9ba-88bf678b4-f4m6w" podStartSLOduration=1.5417351049999999 podStartE2EDuration="1.541735105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:37:24.540633149 +0000 UTC m=+1941.920770794" watchObservedRunningTime="2026-07-28 22:37:24.541735105 +0000 UTC m=+1941.921872748"
Jul 28 22:37:25.519209 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:25.519167    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3db5e9ba-68b64685c-fr4kj" event={"ID":"4a3eb38f-5e6c-4802-bb0a-6b477fe716fb","Type":"ContainerStarted","Data":"9d47f9880e028f7d7234e12ff5cacc02185a23a946caf5d7696cd56fdf3fdb3d"}
Jul 28 22:37:25.538810 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:25.538760    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3db5e9ba-68b64685c-fr4kj" podStartSLOduration=2.088616163 podStartE2EDuration="2.53874477s" totalImagesPullingTime="450.128607ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:37:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:37:25.536395948 +0000 UTC m=+1942.916533593" watchObservedRunningTime="2026-07-28 22:37:25.53874477 +0000 UTC m=+1942.918882417"
Jul 28 22:37:26.524231 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:26.524196    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3db5e9ba-68b64685c-fr4kj"
Jul 28 22:37:32.534153 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:32.534120    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3db5e9ba-68b64685c-fr4kj"
Jul 28 22:37:35.521534 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:35.521498    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-3db5e9ba-88bf678b4-f4m6w"
Jul 28 22:37:35.521892 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:35.521576    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-3db5e9ba-7c94dc4c6b-w25zl"
Jul 28 22:37:43.846330 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:43.846294    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3db5e9ba-68b64685c-fr4kj"]
Jul 28 22:37:43.846828 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:43.846571    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3db5e9ba-68b64685c-fr4kj" podUID="4a3eb38f-5e6c-4802-bb0a-6b477fe716fb" containerName="maas-api" containerID="cri-o://9d47f9880e028f7d7234e12ff5cacc02185a23a946caf5d7696cd56fdf3fdb3d" gracePeriod=30
Jul 28 22:37:43.900126 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:43.900087    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3db5e9ba-7c94dc4c6b-w25zl"]
Jul 28 22:37:43.900537 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:43.900334    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-3db5e9ba-7c94dc4c6b-w25zl" podUID="80c31930-49dd-4e1d-93eb-656693ed0aab" containerName="payload-processing" containerID="cri-o://1e7868d640da2e0d095ab231c0e445c964f59a509e31cb18f3702cd19b0314d6" gracePeriod=30
Jul 28 22:37:43.907731 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:43.907702    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3db5e9ba-88bf678b4-f4m6w"]
Jul 28 22:37:43.908395 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:43.908365    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-3db5e9ba-88bf678b4-f4m6w" podUID="25b9e0e7-f8f0-472b-bc0e-3fc8bfddad36" containerName="payload-pre-processing" containerID="cri-o://8d09c03ec6a12964cf0e8008fa7081a19dd5e4aa9557b05196544bce58f0c017" gracePeriod=30
Jul 28 22:37:44.195610 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.195582    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3db5e9ba-7c94dc4c6b-w25zl"
Jul 28 22:37:44.222149 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.222126    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3db5e9ba-88bf678b4-f4m6w"
Jul 28 22:37:44.226036 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.226007    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3db5e9ba-68b64685c-fr4kj"
Jul 28 22:37:44.278197 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.278166    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80c31930-49dd-4e1d-93eb-656693ed0aab-plugins-config-volume\") pod \"80c31930-49dd-4e1d-93eb-656693ed0aab\" (UID: \"80c31930-49dd-4e1d-93eb-656693ed0aab\") "
Jul 28 22:37:44.278395 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.278253    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwt7j\" (UniqueName: \"kubernetes.io/projected/80c31930-49dd-4e1d-93eb-656693ed0aab-kube-api-access-lwt7j\") pod \"80c31930-49dd-4e1d-93eb-656693ed0aab\" (UID: \"80c31930-49dd-4e1d-93eb-656693ed0aab\") "
Jul 28 22:37:44.278819 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.278744    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80c31930-49dd-4e1d-93eb-656693ed0aab-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "80c31930-49dd-4e1d-93eb-656693ed0aab" (UID: "80c31930-49dd-4e1d-93eb-656693ed0aab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:37:44.280524 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.280500    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80c31930-49dd-4e1d-93eb-656693ed0aab-kube-api-access-lwt7j" (OuterVolumeSpecName: "kube-api-access-lwt7j") pod "80c31930-49dd-4e1d-93eb-656693ed0aab" (UID: "80c31930-49dd-4e1d-93eb-656693ed0aab"). InnerVolumeSpecName "kube-api-access-lwt7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:37:44.379051 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.378961    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a3eb38f-5e6c-4802-bb0a-6b477fe716fb-maas-api-tls\") pod \"4a3eb38f-5e6c-4802-bb0a-6b477fe716fb\" (UID: \"4a3eb38f-5e6c-4802-bb0a-6b477fe716fb\") "
Jul 28 22:37:44.379051 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.379007    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a3eb38f-5e6c-4802-bb0a-6b477fe716fb-openshift-service-ca\") pod \"4a3eb38f-5e6c-4802-bb0a-6b477fe716fb\" (UID: \"4a3eb38f-5e6c-4802-bb0a-6b477fe716fb\") "
Jul 28 22:37:44.379051 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.379039    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c89qs\" (UniqueName: \"kubernetes.io/projected/25b9e0e7-f8f0-472b-bc0e-3fc8bfddad36-kube-api-access-c89qs\") pod \"25b9e0e7-f8f0-472b-bc0e-3fc8bfddad36\" (UID: \"25b9e0e7-f8f0-472b-bc0e-3fc8bfddad36\") "
Jul 28 22:37:44.379341 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.379065    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-df9n5\" (UniqueName: \"kubernetes.io/projected/4a3eb38f-5e6c-4802-bb0a-6b477fe716fb-kube-api-access-df9n5\") pod \"4a3eb38f-5e6c-4802-bb0a-6b477fe716fb\" (UID: \"4a3eb38f-5e6c-4802-bb0a-6b477fe716fb\") "
Jul 28 22:37:44.379341 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.379103    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25b9e0e7-f8f0-472b-bc0e-3fc8bfddad36-plugins-config-volume\") pod \"25b9e0e7-f8f0-472b-bc0e-3fc8bfddad36\" (UID: \"25b9e0e7-f8f0-472b-bc0e-3fc8bfddad36\") "
Jul 28 22:37:44.379484 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.379404    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80c31930-49dd-4e1d-93eb-656693ed0aab-plugins-config-volume\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:37:44.379484 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.379427    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwt7j\" (UniqueName: \"kubernetes.io/projected/80c31930-49dd-4e1d-93eb-656693ed0aab-kube-api-access-lwt7j\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:37:44.379580 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.379512    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a3eb38f-5e6c-4802-bb0a-6b477fe716fb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4a3eb38f-5e6c-4802-bb0a-6b477fe716fb" (UID: "4a3eb38f-5e6c-4802-bb0a-6b477fe716fb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:37:44.379580 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.379564    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25b9e0e7-f8f0-472b-bc0e-3fc8bfddad36-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "25b9e0e7-f8f0-472b-bc0e-3fc8bfddad36" (UID: "25b9e0e7-f8f0-472b-bc0e-3fc8bfddad36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:37:44.381583 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.381537    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a3eb38f-5e6c-4802-bb0a-6b477fe716fb-kube-api-access-df9n5" (OuterVolumeSpecName: "kube-api-access-df9n5") pod "4a3eb38f-5e6c-4802-bb0a-6b477fe716fb" (UID: "4a3eb38f-5e6c-4802-bb0a-6b477fe716fb"). InnerVolumeSpecName "kube-api-access-df9n5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:37:44.381746 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.381726    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25b9e0e7-f8f0-472b-bc0e-3fc8bfddad36-kube-api-access-c89qs" (OuterVolumeSpecName: "kube-api-access-c89qs") pod "25b9e0e7-f8f0-472b-bc0e-3fc8bfddad36" (UID: "25b9e0e7-f8f0-472b-bc0e-3fc8bfddad36"). InnerVolumeSpecName "kube-api-access-c89qs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:37:44.381796 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.381732    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a3eb38f-5e6c-4802-bb0a-6b477fe716fb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4a3eb38f-5e6c-4802-bb0a-6b477fe716fb" (UID: "4a3eb38f-5e6c-4802-bb0a-6b477fe716fb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:37:44.480745 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.480702    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df9n5\" (UniqueName: \"kubernetes.io/projected/4a3eb38f-5e6c-4802-bb0a-6b477fe716fb-kube-api-access-df9n5\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:37:44.480745 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.480730    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25b9e0e7-f8f0-472b-bc0e-3fc8bfddad36-plugins-config-volume\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:37:44.480745 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.480741    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a3eb38f-5e6c-4802-bb0a-6b477fe716fb-maas-api-tls\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:37:44.480745 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.480749    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a3eb38f-5e6c-4802-bb0a-6b477fe716fb-openshift-service-ca\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:37:44.481096 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.480760    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c89qs\" (UniqueName: \"kubernetes.io/projected/25b9e0e7-f8f0-472b-bc0e-3fc8bfddad36-kube-api-access-c89qs\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:37:44.615182 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.615147    2563 generic.go:356] "Generic (PLEG): container finished" podID="4a3eb38f-5e6c-4802-bb0a-6b477fe716fb" containerID="9d47f9880e028f7d7234e12ff5cacc02185a23a946caf5d7696cd56fdf3fdb3d" exitCode=0
Jul 28 22:37:44.615338 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.615189    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3db5e9ba-68b64685c-fr4kj"
Jul 28 22:37:44.615338 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.615232    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3db5e9ba-68b64685c-fr4kj" event={"ID":"4a3eb38f-5e6c-4802-bb0a-6b477fe716fb","Type":"ContainerDied","Data":"9d47f9880e028f7d7234e12ff5cacc02185a23a946caf5d7696cd56fdf3fdb3d"}
Jul 28 22:37:44.615338 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.615282    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3db5e9ba-68b64685c-fr4kj" event={"ID":"4a3eb38f-5e6c-4802-bb0a-6b477fe716fb","Type":"ContainerDied","Data":"b5c5913929c51bfd84eb96c92a17f16635fd969e09f5053caa1856b380bc1c8a"}
Jul 28 22:37:44.615338 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.615306    2563 scope.go:117] "RemoveContainer" containerID="9d47f9880e028f7d7234e12ff5cacc02185a23a946caf5d7696cd56fdf3fdb3d"
Jul 28 22:37:44.616606 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.616583    2563 generic.go:356] "Generic (PLEG): container finished" podID="80c31930-49dd-4e1d-93eb-656693ed0aab" containerID="1e7868d640da2e0d095ab231c0e445c964f59a509e31cb18f3702cd19b0314d6" exitCode=0
Jul 28 22:37:44.616712 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.616620    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3db5e9ba-7c94dc4c6b-w25zl"
Jul 28 22:37:44.616712 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.616627    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3db5e9ba-7c94dc4c6b-w25zl" event={"ID":"80c31930-49dd-4e1d-93eb-656693ed0aab","Type":"ContainerDied","Data":"1e7868d640da2e0d095ab231c0e445c964f59a509e31cb18f3702cd19b0314d6"}
Jul 28 22:37:44.616712 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.616678    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3db5e9ba-7c94dc4c6b-w25zl" event={"ID":"80c31930-49dd-4e1d-93eb-656693ed0aab","Type":"ContainerDied","Data":"b52b08ca9eb9f292b4747d3b588f0b540ca79b4b8711bbc5aa32ac45e79b4ef9"}
Jul 28 22:37:44.618056 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.618032    2563 generic.go:356] "Generic (PLEG): container finished" podID="25b9e0e7-f8f0-472b-bc0e-3fc8bfddad36" containerID="8d09c03ec6a12964cf0e8008fa7081a19dd5e4aa9557b05196544bce58f0c017" exitCode=0
Jul 28 22:37:44.618190 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.618054    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3db5e9ba-88bf678b4-f4m6w" event={"ID":"25b9e0e7-f8f0-472b-bc0e-3fc8bfddad36","Type":"ContainerDied","Data":"8d09c03ec6a12964cf0e8008fa7081a19dd5e4aa9557b05196544bce58f0c017"}
Jul 28 22:37:44.618190 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.618082    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3db5e9ba-88bf678b4-f4m6w"
Jul 28 22:37:44.618190 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.618092    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3db5e9ba-88bf678b4-f4m6w" event={"ID":"25b9e0e7-f8f0-472b-bc0e-3fc8bfddad36","Type":"ContainerDied","Data":"049a907c6a7fdd702a3d73283cd8a44c677efa419c40d85718f5a362762342c9"}
Jul 28 22:37:44.628159 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.627407    2563 scope.go:117] "RemoveContainer" containerID="9d47f9880e028f7d7234e12ff5cacc02185a23a946caf5d7696cd56fdf3fdb3d"
Jul 28 22:37:44.628159 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:37:44.627710    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d47f9880e028f7d7234e12ff5cacc02185a23a946caf5d7696cd56fdf3fdb3d\": container with ID starting with 9d47f9880e028f7d7234e12ff5cacc02185a23a946caf5d7696cd56fdf3fdb3d not found: ID does not exist" containerID="9d47f9880e028f7d7234e12ff5cacc02185a23a946caf5d7696cd56fdf3fdb3d"
Jul 28 22:37:44.628159 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.627745    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d47f9880e028f7d7234e12ff5cacc02185a23a946caf5d7696cd56fdf3fdb3d"} err="failed to get container status \"9d47f9880e028f7d7234e12ff5cacc02185a23a946caf5d7696cd56fdf3fdb3d\": rpc error: code = NotFound desc = could not find container \"9d47f9880e028f7d7234e12ff5cacc02185a23a946caf5d7696cd56fdf3fdb3d\": container with ID starting with 9d47f9880e028f7d7234e12ff5cacc02185a23a946caf5d7696cd56fdf3fdb3d not found: ID does not exist"
Jul 28 22:37:44.628159 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.627777    2563 scope.go:117] "RemoveContainer" containerID="1e7868d640da2e0d095ab231c0e445c964f59a509e31cb18f3702cd19b0314d6"
Jul 28 22:37:44.638740 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.638721    2563 scope.go:117] "RemoveContainer" containerID="1e7868d640da2e0d095ab231c0e445c964f59a509e31cb18f3702cd19b0314d6"
Jul 28 22:37:44.639027 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:37:44.639006    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e7868d640da2e0d095ab231c0e445c964f59a509e31cb18f3702cd19b0314d6\": container with ID starting with 1e7868d640da2e0d095ab231c0e445c964f59a509e31cb18f3702cd19b0314d6 not found: ID does not exist" containerID="1e7868d640da2e0d095ab231c0e445c964f59a509e31cb18f3702cd19b0314d6"
Jul 28 22:37:44.639106 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.639033    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e7868d640da2e0d095ab231c0e445c964f59a509e31cb18f3702cd19b0314d6"} err="failed to get container status \"1e7868d640da2e0d095ab231c0e445c964f59a509e31cb18f3702cd19b0314d6\": rpc error: code = NotFound desc = could not find container \"1e7868d640da2e0d095ab231c0e445c964f59a509e31cb18f3702cd19b0314d6\": container with ID starting with 1e7868d640da2e0d095ab231c0e445c964f59a509e31cb18f3702cd19b0314d6 not found: ID does not exist"
Jul 28 22:37:44.639106 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.639052    2563 scope.go:117] "RemoveContainer" containerID="8d09c03ec6a12964cf0e8008fa7081a19dd5e4aa9557b05196544bce58f0c017"
Jul 28 22:37:44.647072 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.647057    2563 scope.go:117] "RemoveContainer" containerID="8d09c03ec6a12964cf0e8008fa7081a19dd5e4aa9557b05196544bce58f0c017"
Jul 28 22:37:44.647304 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:37:44.647283    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d09c03ec6a12964cf0e8008fa7081a19dd5e4aa9557b05196544bce58f0c017\": container with ID starting with 8d09c03ec6a12964cf0e8008fa7081a19dd5e4aa9557b05196544bce58f0c017 not found: ID does not exist" containerID="8d09c03ec6a12964cf0e8008fa7081a19dd5e4aa9557b05196544bce58f0c017"
Jul 28 22:37:44.647386 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.647315    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d09c03ec6a12964cf0e8008fa7081a19dd5e4aa9557b05196544bce58f0c017"} err="failed to get container status \"8d09c03ec6a12964cf0e8008fa7081a19dd5e4aa9557b05196544bce58f0c017\": rpc error: code = NotFound desc = could not find container \"8d09c03ec6a12964cf0e8008fa7081a19dd5e4aa9557b05196544bce58f0c017\": container with ID starting with 8d09c03ec6a12964cf0e8008fa7081a19dd5e4aa9557b05196544bce58f0c017 not found: ID does not exist"
Jul 28 22:37:44.647729 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.647711    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3db5e9ba-7c94dc4c6b-w25zl"]
Jul 28 22:37:44.653651 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.653630    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3db5e9ba-7c94dc4c6b-w25zl"]
Jul 28 22:37:44.662723 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.662697    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3db5e9ba-68b64685c-fr4kj"]
Jul 28 22:37:44.664794 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.664772    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3db5e9ba-68b64685c-fr4kj"]
Jul 28 22:37:44.675694 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.675674    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3db5e9ba-88bf678b4-f4m6w"]
Jul 28 22:37:44.678953 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:44.678931    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3db5e9ba-88bf678b4-f4m6w"]
Jul 28 22:37:45.131165 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:45.131132    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25b9e0e7-f8f0-472b-bc0e-3fc8bfddad36" path="/var/lib/kubelet/pods/25b9e0e7-f8f0-472b-bc0e-3fc8bfddad36/volumes"
Jul 28 22:37:45.131608 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:45.131591    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a3eb38f-5e6c-4802-bb0a-6b477fe716fb" path="/var/lib/kubelet/pods/4a3eb38f-5e6c-4802-bb0a-6b477fe716fb/volumes"
Jul 28 22:37:45.131928 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:37:45.131917    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80c31930-49dd-4e1d-93eb-656693ed0aab" path="/var/lib/kubelet/pods/80c31930-49dd-4e1d-93eb-656693ed0aab/volumes"
Jul 28 22:39:30.555419 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:30.555311    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cc3d4e93-5ddff9fc48-sv4gv"]
Jul 28 22:39:30.555992 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:30.555737    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a3eb38f-5e6c-4802-bb0a-6b477fe716fb" containerName="maas-api"
Jul 28 22:39:30.555992 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:30.555750    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a3eb38f-5e6c-4802-bb0a-6b477fe716fb" containerName="maas-api"
Jul 28 22:39:30.555992 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:30.555770    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25b9e0e7-f8f0-472b-bc0e-3fc8bfddad36" containerName="payload-pre-processing"
Jul 28 22:39:30.555992 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:30.555776    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="25b9e0e7-f8f0-472b-bc0e-3fc8bfddad36" containerName="payload-pre-processing"
Jul 28 22:39:30.555992 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:30.555787    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80c31930-49dd-4e1d-93eb-656693ed0aab" containerName="payload-processing"
Jul 28 22:39:30.555992 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:30.555793    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="80c31930-49dd-4e1d-93eb-656693ed0aab" containerName="payload-processing"
Jul 28 22:39:30.555992 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:30.555847    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a3eb38f-5e6c-4802-bb0a-6b477fe716fb" containerName="maas-api"
Jul 28 22:39:30.555992 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:30.555855    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="25b9e0e7-f8f0-472b-bc0e-3fc8bfddad36" containerName="payload-pre-processing"
Jul 28 22:39:30.555992 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:30.555861    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="80c31930-49dd-4e1d-93eb-656693ed0aab" containerName="payload-processing"
Jul 28 22:39:30.559037 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:30.559011    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc3d4e93-5ddff9fc48-sv4gv"
Jul 28 22:39:30.561527 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:30.561504    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-cc3d4e93-serving-cert\""
Jul 28 22:39:30.569225 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:30.569199    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cc3d4e93-5ddff9fc48-sv4gv"]
Jul 28 22:39:30.628163 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:30.628130    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0674db86-47cb-421f-8a58-057897c2c595-maas-api-tls\") pod \"maas-api-e2e-mt-cc3d4e93-5ddff9fc48-sv4gv\" (UID: \"0674db86-47cb-421f-8a58-057897c2c595\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc3d4e93-5ddff9fc48-sv4gv"
Jul 28 22:39:30.628337 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:30.628189    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xgh2\" (UniqueName: \"kubernetes.io/projected/0674db86-47cb-421f-8a58-057897c2c595-kube-api-access-9xgh2\") pod \"maas-api-e2e-mt-cc3d4e93-5ddff9fc48-sv4gv\" (UID: \"0674db86-47cb-421f-8a58-057897c2c595\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc3d4e93-5ddff9fc48-sv4gv"
Jul 28 22:39:30.628337 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:30.628307    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0674db86-47cb-421f-8a58-057897c2c595-openshift-service-ca\") pod \"maas-api-e2e-mt-cc3d4e93-5ddff9fc48-sv4gv\" (UID: \"0674db86-47cb-421f-8a58-057897c2c595\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc3d4e93-5ddff9fc48-sv4gv"
Jul 28 22:39:30.729564 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:30.729526    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xgh2\" (UniqueName: \"kubernetes.io/projected/0674db86-47cb-421f-8a58-057897c2c595-kube-api-access-9xgh2\") pod \"maas-api-e2e-mt-cc3d4e93-5ddff9fc48-sv4gv\" (UID: \"0674db86-47cb-421f-8a58-057897c2c595\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc3d4e93-5ddff9fc48-sv4gv"
Jul 28 22:39:30.729769 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:30.729645    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0674db86-47cb-421f-8a58-057897c2c595-openshift-service-ca\") pod \"maas-api-e2e-mt-cc3d4e93-5ddff9fc48-sv4gv\" (UID: \"0674db86-47cb-421f-8a58-057897c2c595\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc3d4e93-5ddff9fc48-sv4gv"
Jul 28 22:39:30.729769 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:30.729676    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0674db86-47cb-421f-8a58-057897c2c595-maas-api-tls\") pod \"maas-api-e2e-mt-cc3d4e93-5ddff9fc48-sv4gv\" (UID: \"0674db86-47cb-421f-8a58-057897c2c595\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc3d4e93-5ddff9fc48-sv4gv"
Jul 28 22:39:30.730334 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:30.730308    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0674db86-47cb-421f-8a58-057897c2c595-openshift-service-ca\") pod \"maas-api-e2e-mt-cc3d4e93-5ddff9fc48-sv4gv\" (UID: \"0674db86-47cb-421f-8a58-057897c2c595\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc3d4e93-5ddff9fc48-sv4gv"
Jul 28 22:39:30.732057 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:30.732030    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0674db86-47cb-421f-8a58-057897c2c595-maas-api-tls\") pod \"maas-api-e2e-mt-cc3d4e93-5ddff9fc48-sv4gv\" (UID: \"0674db86-47cb-421f-8a58-057897c2c595\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc3d4e93-5ddff9fc48-sv4gv"
Jul 28 22:39:30.782895 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:30.782865    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xgh2\" (UniqueName: \"kubernetes.io/projected/0674db86-47cb-421f-8a58-057897c2c595-kube-api-access-9xgh2\") pod \"maas-api-e2e-mt-cc3d4e93-5ddff9fc48-sv4gv\" (UID: \"0674db86-47cb-421f-8a58-057897c2c595\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc3d4e93-5ddff9fc48-sv4gv"
Jul 28 22:39:30.869558 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:30.869479    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc3d4e93-5ddff9fc48-sv4gv"
Jul 28 22:39:31.020718 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:31.020695    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cc3d4e93-5ddff9fc48-sv4gv"]
Jul 28 22:39:31.022296 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:39:31.022270    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0674db86_47cb_421f_8a58_057897c2c595.slice/crio-fcbc43e0798530b7071b371ee666a2ccfb6d13e8f8ea0fe5f58dbd8cf7b36996 WatchSource:0}: Error finding container fcbc43e0798530b7071b371ee666a2ccfb6d13e8f8ea0fe5f58dbd8cf7b36996: Status 404 returned error can't find the container with id fcbc43e0798530b7071b371ee666a2ccfb6d13e8f8ea0fe5f58dbd8cf7b36996
Jul 28 22:39:31.106024 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:31.105988    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc3d4e93-5ddff9fc48-sv4gv" event={"ID":"0674db86-47cb-421f-8a58-057897c2c595","Type":"ContainerStarted","Data":"fcbc43e0798530b7071b371ee666a2ccfb6d13e8f8ea0fe5f58dbd8cf7b36996"}
Jul 28 22:39:31.565923 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:31.565888    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cc3d4e93-7955c97dd4-sk296"]
Jul 28 22:39:31.569337 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:31.569318    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cc3d4e93-7955c97dd4-sk296"
Jul 28 22:39:31.573395 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:31.573293    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-cc3d4e93-dockercfg-g44lt\""
Jul 28 22:39:31.573519 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:31.573476    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-cc3d4e93\""
Jul 28 22:39:31.586491 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:31.586468    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cc3d4e93-7955c97dd4-sk296"]
Jul 28 22:39:31.637606 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:31.637571    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g695x\" (UniqueName: \"kubernetes.io/projected/6d26dec9-f6b1-405d-a7c7-1f3838fe104f-kube-api-access-g695x\") pod \"payload-pre-processing-e2e-mt-cc3d4e93-7955c97dd4-sk296\" (UID: \"6d26dec9-f6b1-405d-a7c7-1f3838fe104f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cc3d4e93-7955c97dd4-sk296"
Jul 28 22:39:31.637770 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:31.637610    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d26dec9-f6b1-405d-a7c7-1f3838fe104f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cc3d4e93-7955c97dd4-sk296\" (UID: \"6d26dec9-f6b1-405d-a7c7-1f3838fe104f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cc3d4e93-7955c97dd4-sk296"
Jul 28 22:39:31.694592 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:31.694544    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cc3d4e93-777594c69d-n487f"]
Jul 28 22:39:31.697880 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:31.697862    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cc3d4e93-777594c69d-n487f"
Jul 28 22:39:31.714993 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:31.714967    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cc3d4e93-777594c69d-n487f"]
Jul 28 22:39:31.738724 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:31.738480    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g695x\" (UniqueName: \"kubernetes.io/projected/6d26dec9-f6b1-405d-a7c7-1f3838fe104f-kube-api-access-g695x\") pod \"payload-pre-processing-e2e-mt-cc3d4e93-7955c97dd4-sk296\" (UID: \"6d26dec9-f6b1-405d-a7c7-1f3838fe104f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cc3d4e93-7955c97dd4-sk296"
Jul 28 22:39:31.738966 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:31.738943    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d26dec9-f6b1-405d-a7c7-1f3838fe104f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cc3d4e93-7955c97dd4-sk296\" (UID: \"6d26dec9-f6b1-405d-a7c7-1f3838fe104f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cc3d4e93-7955c97dd4-sk296"
Jul 28 22:39:31.739119 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:31.739102    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5mlt\" (UniqueName: \"kubernetes.io/projected/4b1e9b0b-7e44-4715-8c2b-ccdb74eaa052-kube-api-access-s5mlt\") pod \"payload-processing-e2e-mt-cc3d4e93-777594c69d-n487f\" (UID: \"4b1e9b0b-7e44-4715-8c2b-ccdb74eaa052\") " pod="openshift-ingress/payload-processing-e2e-mt-cc3d4e93-777594c69d-n487f"
Jul 28 22:39:31.739320 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:31.739301    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b1e9b0b-7e44-4715-8c2b-ccdb74eaa052-plugins-config-volume\") pod \"payload-processing-e2e-mt-cc3d4e93-777594c69d-n487f\" (UID: \"4b1e9b0b-7e44-4715-8c2b-ccdb74eaa052\") " pod="openshift-ingress/payload-processing-e2e-mt-cc3d4e93-777594c69d-n487f"
Jul 28 22:39:31.739988 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:31.739967    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d26dec9-f6b1-405d-a7c7-1f3838fe104f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cc3d4e93-7955c97dd4-sk296\" (UID: \"6d26dec9-f6b1-405d-a7c7-1f3838fe104f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cc3d4e93-7955c97dd4-sk296"
Jul 28 22:39:31.747857 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:31.747834    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g695x\" (UniqueName: \"kubernetes.io/projected/6d26dec9-f6b1-405d-a7c7-1f3838fe104f-kube-api-access-g695x\") pod \"payload-pre-processing-e2e-mt-cc3d4e93-7955c97dd4-sk296\" (UID: \"6d26dec9-f6b1-405d-a7c7-1f3838fe104f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cc3d4e93-7955c97dd4-sk296"
Jul 28 22:39:31.840764 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:31.840724    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5mlt\" (UniqueName: \"kubernetes.io/projected/4b1e9b0b-7e44-4715-8c2b-ccdb74eaa052-kube-api-access-s5mlt\") pod \"payload-processing-e2e-mt-cc3d4e93-777594c69d-n487f\" (UID: \"4b1e9b0b-7e44-4715-8c2b-ccdb74eaa052\") " pod="openshift-ingress/payload-processing-e2e-mt-cc3d4e93-777594c69d-n487f"
Jul 28 22:39:31.840944 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:31.840822    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b1e9b0b-7e44-4715-8c2b-ccdb74eaa052-plugins-config-volume\") pod \"payload-processing-e2e-mt-cc3d4e93-777594c69d-n487f\" (UID: \"4b1e9b0b-7e44-4715-8c2b-ccdb74eaa052\") " pod="openshift-ingress/payload-processing-e2e-mt-cc3d4e93-777594c69d-n487f"
Jul 28 22:39:31.841644 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:31.841620    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b1e9b0b-7e44-4715-8c2b-ccdb74eaa052-plugins-config-volume\") pod \"payload-processing-e2e-mt-cc3d4e93-777594c69d-n487f\" (UID: \"4b1e9b0b-7e44-4715-8c2b-ccdb74eaa052\") " pod="openshift-ingress/payload-processing-e2e-mt-cc3d4e93-777594c69d-n487f"
Jul 28 22:39:31.850865 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:31.850839    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5mlt\" (UniqueName: \"kubernetes.io/projected/4b1e9b0b-7e44-4715-8c2b-ccdb74eaa052-kube-api-access-s5mlt\") pod \"payload-processing-e2e-mt-cc3d4e93-777594c69d-n487f\" (UID: \"4b1e9b0b-7e44-4715-8c2b-ccdb74eaa052\") " pod="openshift-ingress/payload-processing-e2e-mt-cc3d4e93-777594c69d-n487f"
Jul 28 22:39:31.880465 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:31.880439    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cc3d4e93-7955c97dd4-sk296"
Jul 28 22:39:32.009425 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:32.009389    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cc3d4e93-777594c69d-n487f"
Jul 28 22:39:32.022445 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:32.022419    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cc3d4e93-7955c97dd4-sk296"]
Jul 28 22:39:32.024410 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:39:32.024384    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d26dec9_f6b1_405d_a7c7_1f3838fe104f.slice/crio-fb4840d2585ee062b271753785117e314e5743e14b1fd15ba0937a48872ec055 WatchSource:0}: Error finding container fb4840d2585ee062b271753785117e314e5743e14b1fd15ba0937a48872ec055: Status 404 returned error can't find the container with id fb4840d2585ee062b271753785117e314e5743e14b1fd15ba0937a48872ec055
Jul 28 22:39:32.114772 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:32.114387    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc3d4e93-5ddff9fc48-sv4gv" event={"ID":"0674db86-47cb-421f-8a58-057897c2c595","Type":"ContainerStarted","Data":"a34342fb3049a13a6381ddbe0590b9bc80958c6086e1516e5b52e3584c7136a9"}
Jul 28 22:39:32.114772 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:32.114619    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc3d4e93-5ddff9fc48-sv4gv"
Jul 28 22:39:32.116172 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:32.116152    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cc3d4e93-7955c97dd4-sk296" event={"ID":"6d26dec9-f6b1-405d-a7c7-1f3838fe104f","Type":"ContainerStarted","Data":"29288a3a726b8e5d0a6061138694aeff657cfca3322fb3dadc35708ae15353c9"}
Jul 28 22:39:32.116267 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:32.116188    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cc3d4e93-7955c97dd4-sk296" event={"ID":"6d26dec9-f6b1-405d-a7c7-1f3838fe104f","Type":"ContainerStarted","Data":"fb4840d2585ee062b271753785117e314e5743e14b1fd15ba0937a48872ec055"}
Jul 28 22:39:32.116267 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:32.116220    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-cc3d4e93-7955c97dd4-sk296"
Jul 28 22:39:32.137550 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:32.137385    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc3d4e93-5ddff9fc48-sv4gv" podStartSLOduration=1.738050665 podStartE2EDuration="2.137364946s" totalImagesPullingTime="399.314281ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:39:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:39:32.134840492 +0000 UTC m=+2069.514978136" watchObservedRunningTime="2026-07-28 22:39:32.137364946 +0000 UTC m=+2069.517502598"
Jul 28 22:39:32.152447 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:32.152410    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cc3d4e93-777594c69d-n487f"]
Jul 28 22:39:32.156137 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:39:32.156112    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b1e9b0b_7e44_4715_8c2b_ccdb74eaa052.slice/crio-92c662de52833c3207899a72921895cf13918421814705c07feb2c968d9e961a WatchSource:0}: Error finding container 92c662de52833c3207899a72921895cf13918421814705c07feb2c968d9e961a: Status 404 returned error can't find the container with id 92c662de52833c3207899a72921895cf13918421814705c07feb2c968d9e961a
Jul 28 22:39:32.158760 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:32.158718    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-cc3d4e93-7955c97dd4-sk296" podStartSLOduration=1.158701559 podStartE2EDuration="1.158701559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:39:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:39:32.156822795 +0000 UTC m=+2069.536960440" watchObservedRunningTime="2026-07-28 22:39:32.158701559 +0000 UTC m=+2069.538839202"
Jul 28 22:39:33.122449 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:33.122412    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cc3d4e93-777594c69d-n487f" event={"ID":"4b1e9b0b-7e44-4715-8c2b-ccdb74eaa052","Type":"ContainerStarted","Data":"d47a7a23446e2ea8e5846f630da8ed7aec19067c2efc379b74b3aad5955046ff"}
Jul 28 22:39:33.122836 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:33.122473    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cc3d4e93-777594c69d-n487f" event={"ID":"4b1e9b0b-7e44-4715-8c2b-ccdb74eaa052","Type":"ContainerStarted","Data":"92c662de52833c3207899a72921895cf13918421814705c07feb2c968d9e961a"}
Jul 28 22:39:33.122836 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:33.122628    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-cc3d4e93-777594c69d-n487f"
Jul 28 22:39:33.167010 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:33.166963    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-cc3d4e93-777594c69d-n487f" podStartSLOduration=2.166948572 podStartE2EDuration="2.166948572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:39:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:39:33.164166534 +0000 UTC m=+2070.544304177" watchObservedRunningTime="2026-07-28 22:39:33.166948572 +0000 UTC m=+2070.547086216"
Jul 28 22:39:38.127817 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:38.127782    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc3d4e93-5ddff9fc48-sv4gv"
Jul 28 22:39:43.131068 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:43.131039    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-cc3d4e93-7955c97dd4-sk296"
Jul 28 22:39:44.128769 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:39:44.128738    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-cc3d4e93-777594c69d-n487f"
Jul 28 22:40:03.152720 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:03.152691    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mcj8p_7bab32b0-13c6-4055-916e-a7454bafb5b6/ovn-acl-logging/0.log"
Jul 28 22:40:03.154523 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:03.154500    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mcj8p_7bab32b0-13c6-4055-916e-a7454bafb5b6/ovn-acl-logging/0.log"
Jul 28 22:40:12.870264 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:12.870230    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cc3d4e93-5ddff9fc48-sv4gv"]
Jul 28 22:40:12.870812 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:12.870476    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc3d4e93-5ddff9fc48-sv4gv" podUID="0674db86-47cb-421f-8a58-057897c2c595" containerName="maas-api" containerID="cri-o://a34342fb3049a13a6381ddbe0590b9bc80958c6086e1516e5b52e3584c7136a9" gracePeriod=30
Jul 28 22:40:12.926649 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:12.926613    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cc3d4e93-777594c69d-n487f"]
Jul 28 22:40:12.926889 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:12.926859    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-cc3d4e93-777594c69d-n487f" podUID="4b1e9b0b-7e44-4715-8c2b-ccdb74eaa052" containerName="payload-processing" containerID="cri-o://d47a7a23446e2ea8e5846f630da8ed7aec19067c2efc379b74b3aad5955046ff" gracePeriod=30
Jul 28 22:40:12.932098 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:12.931700    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cc3d4e93-7955c97dd4-sk296"]
Jul 28 22:40:12.932098 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:12.931927    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-cc3d4e93-7955c97dd4-sk296" podUID="6d26dec9-f6b1-405d-a7c7-1f3838fe104f" containerName="payload-pre-processing" containerID="cri-o://29288a3a726b8e5d0a6061138694aeff657cfca3322fb3dadc35708ae15353c9" gracePeriod=30
Jul 28 22:40:13.123380 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.123214    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-cc3d4e93-7955c97dd4-sk296" podUID="6d26dec9-f6b1-405d-a7c7-1f3838fe104f" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.57:9004: connect: connection refused"
Jul 28 22:40:13.179057 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.179029    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc3d4e93-5ddff9fc48-sv4gv"
Jul 28 22:40:13.209651 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.209628    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cc3d4e93-777594c69d-n487f"
Jul 28 22:40:13.214998 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.214942    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xgh2\" (UniqueName: \"kubernetes.io/projected/0674db86-47cb-421f-8a58-057897c2c595-kube-api-access-9xgh2\") pod \"0674db86-47cb-421f-8a58-057897c2c595\" (UID: \"0674db86-47cb-421f-8a58-057897c2c595\") "
Jul 28 22:40:13.215123 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.215048    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0674db86-47cb-421f-8a58-057897c2c595-openshift-service-ca\") pod \"0674db86-47cb-421f-8a58-057897c2c595\" (UID: \"0674db86-47cb-421f-8a58-057897c2c595\") "
Jul 28 22:40:13.215201 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.215141    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0674db86-47cb-421f-8a58-057897c2c595-maas-api-tls\") pod \"0674db86-47cb-421f-8a58-057897c2c595\" (UID: \"0674db86-47cb-421f-8a58-057897c2c595\") "
Jul 28 22:40:13.215535 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.215508    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0674db86-47cb-421f-8a58-057897c2c595-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0674db86-47cb-421f-8a58-057897c2c595" (UID: "0674db86-47cb-421f-8a58-057897c2c595"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:40:13.217104 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.217075    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0674db86-47cb-421f-8a58-057897c2c595-kube-api-access-9xgh2" (OuterVolumeSpecName: "kube-api-access-9xgh2") pod "0674db86-47cb-421f-8a58-057897c2c595" (UID: "0674db86-47cb-421f-8a58-057897c2c595"). InnerVolumeSpecName "kube-api-access-9xgh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:40:13.217319 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.217285    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0674db86-47cb-421f-8a58-057897c2c595-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0674db86-47cb-421f-8a58-057897c2c595" (UID: "0674db86-47cb-421f-8a58-057897c2c595"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:40:13.218800 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.218784    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cc3d4e93-7955c97dd4-sk296"
Jul 28 22:40:13.314785 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.314747    2563 generic.go:356] "Generic (PLEG): container finished" podID="4b1e9b0b-7e44-4715-8c2b-ccdb74eaa052" containerID="d47a7a23446e2ea8e5846f630da8ed7aec19067c2efc379b74b3aad5955046ff" exitCode=0
Jul 28 22:40:13.314964 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.314796    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cc3d4e93-777594c69d-n487f"
Jul 28 22:40:13.314964 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.314834    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cc3d4e93-777594c69d-n487f" event={"ID":"4b1e9b0b-7e44-4715-8c2b-ccdb74eaa052","Type":"ContainerDied","Data":"d47a7a23446e2ea8e5846f630da8ed7aec19067c2efc379b74b3aad5955046ff"}
Jul 28 22:40:13.314964 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.314887    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cc3d4e93-777594c69d-n487f" event={"ID":"4b1e9b0b-7e44-4715-8c2b-ccdb74eaa052","Type":"ContainerDied","Data":"92c662de52833c3207899a72921895cf13918421814705c07feb2c968d9e961a"}
Jul 28 22:40:13.314964 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.314916    2563 scope.go:117] "RemoveContainer" containerID="d47a7a23446e2ea8e5846f630da8ed7aec19067c2efc379b74b3aad5955046ff"
Jul 28 22:40:13.315691 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.315666    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5mlt\" (UniqueName: \"kubernetes.io/projected/4b1e9b0b-7e44-4715-8c2b-ccdb74eaa052-kube-api-access-s5mlt\") pod \"4b1e9b0b-7e44-4715-8c2b-ccdb74eaa052\" (UID: \"4b1e9b0b-7e44-4715-8c2b-ccdb74eaa052\") "
Jul 28 22:40:13.315836 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.315703    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b1e9b0b-7e44-4715-8c2b-ccdb74eaa052-plugins-config-volume\") pod \"4b1e9b0b-7e44-4715-8c2b-ccdb74eaa052\" (UID: \"4b1e9b0b-7e44-4715-8c2b-ccdb74eaa052\") "
Jul 28 22:40:13.315836 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.315744    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d26dec9-f6b1-405d-a7c7-1f3838fe104f-plugins-config-volume\") pod \"6d26dec9-f6b1-405d-a7c7-1f3838fe104f\" (UID: \"6d26dec9-f6b1-405d-a7c7-1f3838fe104f\") "
Jul 28 22:40:13.315836 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.315763    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g695x\" (UniqueName: \"kubernetes.io/projected/6d26dec9-f6b1-405d-a7c7-1f3838fe104f-kube-api-access-g695x\") pod \"6d26dec9-f6b1-405d-a7c7-1f3838fe104f\" (UID: \"6d26dec9-f6b1-405d-a7c7-1f3838fe104f\") "
Jul 28 22:40:13.316076 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.316058    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0674db86-47cb-421f-8a58-057897c2c595-maas-api-tls\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:40:13.316212 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.316078    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xgh2\" (UniqueName: \"kubernetes.io/projected/0674db86-47cb-421f-8a58-057897c2c595-kube-api-access-9xgh2\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:40:13.316212 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.316094    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0674db86-47cb-421f-8a58-057897c2c595-openshift-service-ca\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:40:13.316212 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.316154    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b1e9b0b-7e44-4715-8c2b-ccdb74eaa052-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4b1e9b0b-7e44-4715-8c2b-ccdb74eaa052" (UID: "4b1e9b0b-7e44-4715-8c2b-ccdb74eaa052"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:40:13.316212 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.316175    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d26dec9-f6b1-405d-a7c7-1f3838fe104f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6d26dec9-f6b1-405d-a7c7-1f3838fe104f" (UID: "6d26dec9-f6b1-405d-a7c7-1f3838fe104f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:40:13.316530 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.316502    2563 generic.go:356] "Generic (PLEG): container finished" podID="6d26dec9-f6b1-405d-a7c7-1f3838fe104f" containerID="29288a3a726b8e5d0a6061138694aeff657cfca3322fb3dadc35708ae15353c9" exitCode=0
Jul 28 22:40:13.316640 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.316545    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cc3d4e93-7955c97dd4-sk296"
Jul 28 22:40:13.316640 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.316611    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cc3d4e93-7955c97dd4-sk296" event={"ID":"6d26dec9-f6b1-405d-a7c7-1f3838fe104f","Type":"ContainerDied","Data":"29288a3a726b8e5d0a6061138694aeff657cfca3322fb3dadc35708ae15353c9"}
Jul 28 22:40:13.316731 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.316668    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cc3d4e93-7955c97dd4-sk296" event={"ID":"6d26dec9-f6b1-405d-a7c7-1f3838fe104f","Type":"ContainerDied","Data":"fb4840d2585ee062b271753785117e314e5743e14b1fd15ba0937a48872ec055"}
Jul 28 22:40:13.318218 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.318135    2563 generic.go:356] "Generic (PLEG): container finished" podID="0674db86-47cb-421f-8a58-057897c2c595" containerID="a34342fb3049a13a6381ddbe0590b9bc80958c6086e1516e5b52e3584c7136a9" exitCode=0
Jul 28 22:40:13.318218 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.318163    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc3d4e93-5ddff9fc48-sv4gv" event={"ID":"0674db86-47cb-421f-8a58-057897c2c595","Type":"ContainerDied","Data":"a34342fb3049a13a6381ddbe0590b9bc80958c6086e1516e5b52e3584c7136a9"}
Jul 28 22:40:13.318218 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.318173    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b1e9b0b-7e44-4715-8c2b-ccdb74eaa052-kube-api-access-s5mlt" (OuterVolumeSpecName: "kube-api-access-s5mlt") pod "4b1e9b0b-7e44-4715-8c2b-ccdb74eaa052" (UID: "4b1e9b0b-7e44-4715-8c2b-ccdb74eaa052"). InnerVolumeSpecName "kube-api-access-s5mlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:40:13.318218 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.318210    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc3d4e93-5ddff9fc48-sv4gv" event={"ID":"0674db86-47cb-421f-8a58-057897c2c595","Type":"ContainerDied","Data":"fcbc43e0798530b7071b371ee666a2ccfb6d13e8f8ea0fe5f58dbd8cf7b36996"}
Jul 28 22:40:13.318517 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.318174    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc3d4e93-5ddff9fc48-sv4gv"
Jul 28 22:40:13.318517 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.318243    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d26dec9-f6b1-405d-a7c7-1f3838fe104f-kube-api-access-g695x" (OuterVolumeSpecName: "kube-api-access-g695x") pod "6d26dec9-f6b1-405d-a7c7-1f3838fe104f" (UID: "6d26dec9-f6b1-405d-a7c7-1f3838fe104f"). InnerVolumeSpecName "kube-api-access-g695x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:40:13.323748 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.323729    2563 scope.go:117] "RemoveContainer" containerID="d47a7a23446e2ea8e5846f630da8ed7aec19067c2efc379b74b3aad5955046ff"
Jul 28 22:40:13.323985 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:40:13.323963    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d47a7a23446e2ea8e5846f630da8ed7aec19067c2efc379b74b3aad5955046ff\": container with ID starting with d47a7a23446e2ea8e5846f630da8ed7aec19067c2efc379b74b3aad5955046ff not found: ID does not exist" containerID="d47a7a23446e2ea8e5846f630da8ed7aec19067c2efc379b74b3aad5955046ff"
Jul 28 22:40:13.324070 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.323995    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d47a7a23446e2ea8e5846f630da8ed7aec19067c2efc379b74b3aad5955046ff"} err="failed to get container status \"d47a7a23446e2ea8e5846f630da8ed7aec19067c2efc379b74b3aad5955046ff\": rpc error: code = NotFound desc = could not find container \"d47a7a23446e2ea8e5846f630da8ed7aec19067c2efc379b74b3aad5955046ff\": container with ID starting with d47a7a23446e2ea8e5846f630da8ed7aec19067c2efc379b74b3aad5955046ff not found: ID does not exist"
Jul 28 22:40:13.324070 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.324018    2563 scope.go:117] "RemoveContainer" containerID="29288a3a726b8e5d0a6061138694aeff657cfca3322fb3dadc35708ae15353c9"
Jul 28 22:40:13.331563 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.331546    2563 scope.go:117] "RemoveContainer" containerID="29288a3a726b8e5d0a6061138694aeff657cfca3322fb3dadc35708ae15353c9"
Jul 28 22:40:13.331804 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:40:13.331774    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29288a3a726b8e5d0a6061138694aeff657cfca3322fb3dadc35708ae15353c9\": container with ID starting with 29288a3a726b8e5d0a6061138694aeff657cfca3322fb3dadc35708ae15353c9 not found: ID does not exist" containerID="29288a3a726b8e5d0a6061138694aeff657cfca3322fb3dadc35708ae15353c9"
Jul 28 22:40:13.331849 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.331811    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29288a3a726b8e5d0a6061138694aeff657cfca3322fb3dadc35708ae15353c9"} err="failed to get container status \"29288a3a726b8e5d0a6061138694aeff657cfca3322fb3dadc35708ae15353c9\": rpc error: code = NotFound desc = could not find container \"29288a3a726b8e5d0a6061138694aeff657cfca3322fb3dadc35708ae15353c9\": container with ID starting with 29288a3a726b8e5d0a6061138694aeff657cfca3322fb3dadc35708ae15353c9 not found: ID does not exist"
Jul 28 22:40:13.331849 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.331827    2563 scope.go:117] "RemoveContainer" containerID="a34342fb3049a13a6381ddbe0590b9bc80958c6086e1516e5b52e3584c7136a9"
Jul 28 22:40:13.339306 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.339283    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cc3d4e93-5ddff9fc48-sv4gv"]
Jul 28 22:40:13.339986 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.339968    2563 scope.go:117] "RemoveContainer" containerID="a34342fb3049a13a6381ddbe0590b9bc80958c6086e1516e5b52e3584c7136a9"
Jul 28 22:40:13.340336 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:40:13.340304    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a34342fb3049a13a6381ddbe0590b9bc80958c6086e1516e5b52e3584c7136a9\": container with ID starting with a34342fb3049a13a6381ddbe0590b9bc80958c6086e1516e5b52e3584c7136a9 not found: ID does not exist" containerID="a34342fb3049a13a6381ddbe0590b9bc80958c6086e1516e5b52e3584c7136a9"
Jul 28 22:40:13.340502 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.340335    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a34342fb3049a13a6381ddbe0590b9bc80958c6086e1516e5b52e3584c7136a9"} err="failed to get container status \"a34342fb3049a13a6381ddbe0590b9bc80958c6086e1516e5b52e3584c7136a9\": rpc error: code = NotFound desc = could not find container \"a34342fb3049a13a6381ddbe0590b9bc80958c6086e1516e5b52e3584c7136a9\": container with ID starting with a34342fb3049a13a6381ddbe0590b9bc80958c6086e1516e5b52e3584c7136a9 not found: ID does not exist"
Jul 28 22:40:13.342685 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.342665    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cc3d4e93-5ddff9fc48-sv4gv"]
Jul 28 22:40:13.417039 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.416956    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5mlt\" (UniqueName: \"kubernetes.io/projected/4b1e9b0b-7e44-4715-8c2b-ccdb74eaa052-kube-api-access-s5mlt\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:40:13.417039 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.416983    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b1e9b0b-7e44-4715-8c2b-ccdb74eaa052-plugins-config-volume\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:40:13.417039 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.416997    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d26dec9-f6b1-405d-a7c7-1f3838fe104f-plugins-config-volume\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:40:13.417039 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.417009    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g695x\" (UniqueName: \"kubernetes.io/projected/6d26dec9-f6b1-405d-a7c7-1f3838fe104f-kube-api-access-g695x\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:40:13.639286 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.639235    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cc3d4e93-7955c97dd4-sk296"]
Jul 28 22:40:13.641261 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.641235    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cc3d4e93-7955c97dd4-sk296"]
Jul 28 22:40:13.651791 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.651765    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cc3d4e93-777594c69d-n487f"]
Jul 28 22:40:13.653459 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:13.653437    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cc3d4e93-777594c69d-n487f"]
Jul 28 22:40:15.130702 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:15.130666    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0674db86-47cb-421f-8a58-057897c2c595" path="/var/lib/kubelet/pods/0674db86-47cb-421f-8a58-057897c2c595/volumes"
Jul 28 22:40:15.131075 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:15.131020    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b1e9b0b-7e44-4715-8c2b-ccdb74eaa052" path="/var/lib/kubelet/pods/4b1e9b0b-7e44-4715-8c2b-ccdb74eaa052/volumes"
Jul 28 22:40:15.131330 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:15.131317    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d26dec9-f6b1-405d-a7c7-1f3838fe104f" path="/var/lib/kubelet/pods/6d26dec9-f6b1-405d-a7c7-1f3838fe104f/volumes"
Jul 28 22:40:16.123507 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:16.123464    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc3d4e93-5ddff9fc48-sv4gv" podUID="0674db86-47cb-421f-8a58-057897c2c595" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.56:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 28 22:40:42.648934 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:42.648898    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25c13375-7fb5867997-ggrbl"]
Jul 28 22:40:42.649567 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:42.649546    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d26dec9-f6b1-405d-a7c7-1f3838fe104f" containerName="payload-pre-processing"
Jul 28 22:40:42.649646 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:42.649570    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d26dec9-f6b1-405d-a7c7-1f3838fe104f" containerName="payload-pre-processing"
Jul 28 22:40:42.649646 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:42.649603    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0674db86-47cb-421f-8a58-057897c2c595" containerName="maas-api"
Jul 28 22:40:42.649646 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:42.649613    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0674db86-47cb-421f-8a58-057897c2c595" containerName="maas-api"
Jul 28 22:40:42.649646 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:42.649633    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b1e9b0b-7e44-4715-8c2b-ccdb74eaa052" containerName="payload-processing"
Jul 28 22:40:42.649646 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:42.649642    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b1e9b0b-7e44-4715-8c2b-ccdb74eaa052" containerName="payload-processing"
Jul 28 22:40:42.649892 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:42.649732    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d26dec9-f6b1-405d-a7c7-1f3838fe104f" containerName="payload-pre-processing"
Jul 28 22:40:42.649892 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:42.649745    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0674db86-47cb-421f-8a58-057897c2c595" containerName="maas-api"
Jul 28 22:40:42.649892 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:42.649755    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b1e9b0b-7e44-4715-8c2b-ccdb74eaa052" containerName="payload-processing"
Jul 28 22:40:42.652860 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:42.652839    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25c13375-7fb5867997-ggrbl"
Jul 28 22:40:42.655385 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:42.655344    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-25c13375-serving-cert\""
Jul 28 22:40:42.663573 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:42.663549    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25c13375-7fb5867997-ggrbl"]
Jul 28 22:40:42.783223 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:42.783182    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5952\" (UniqueName: \"kubernetes.io/projected/c2078f8d-9360-43eb-9f01-7b690f07a53c-kube-api-access-t5952\") pod \"maas-api-e2e-mt-25c13375-7fb5867997-ggrbl\" (UID: \"c2078f8d-9360-43eb-9f01-7b690f07a53c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25c13375-7fb5867997-ggrbl"
Jul 28 22:40:42.783402 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:42.783305    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2078f8d-9360-43eb-9f01-7b690f07a53c-maas-api-tls\") pod \"maas-api-e2e-mt-25c13375-7fb5867997-ggrbl\" (UID: \"c2078f8d-9360-43eb-9f01-7b690f07a53c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25c13375-7fb5867997-ggrbl"
Jul 28 22:40:42.783402 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:42.783339    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2078f8d-9360-43eb-9f01-7b690f07a53c-openshift-service-ca\") pod \"maas-api-e2e-mt-25c13375-7fb5867997-ggrbl\" (UID: \"c2078f8d-9360-43eb-9f01-7b690f07a53c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25c13375-7fb5867997-ggrbl"
Jul 28 22:40:42.884491 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:42.884457    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2078f8d-9360-43eb-9f01-7b690f07a53c-maas-api-tls\") pod \"maas-api-e2e-mt-25c13375-7fb5867997-ggrbl\" (UID: \"c2078f8d-9360-43eb-9f01-7b690f07a53c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25c13375-7fb5867997-ggrbl"
Jul 28 22:40:42.884491 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:42.884490    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2078f8d-9360-43eb-9f01-7b690f07a53c-openshift-service-ca\") pod \"maas-api-e2e-mt-25c13375-7fb5867997-ggrbl\" (UID: \"c2078f8d-9360-43eb-9f01-7b690f07a53c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25c13375-7fb5867997-ggrbl"
Jul 28 22:40:42.884708 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:42.884536    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5952\" (UniqueName: \"kubernetes.io/projected/c2078f8d-9360-43eb-9f01-7b690f07a53c-kube-api-access-t5952\") pod \"maas-api-e2e-mt-25c13375-7fb5867997-ggrbl\" (UID: \"c2078f8d-9360-43eb-9f01-7b690f07a53c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25c13375-7fb5867997-ggrbl"
Jul 28 22:40:42.884708 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:40:42.884620    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-25c13375-serving-cert: secret "maas-api-e2e-mt-25c13375-serving-cert" not found
Jul 28 22:40:42.884786 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:40:42.884709    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2078f8d-9360-43eb-9f01-7b690f07a53c-maas-api-tls podName:c2078f8d-9360-43eb-9f01-7b690f07a53c nodeName:}" failed. No retries permitted until 2026-07-28 22:40:43.384691732 +0000 UTC m=+2140.764829374 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c2078f8d-9360-43eb-9f01-7b690f07a53c-maas-api-tls") pod "maas-api-e2e-mt-25c13375-7fb5867997-ggrbl" (UID: "c2078f8d-9360-43eb-9f01-7b690f07a53c") : secret "maas-api-e2e-mt-25c13375-serving-cert" not found
Jul 28 22:40:42.885230 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:42.885213    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2078f8d-9360-43eb-9f01-7b690f07a53c-openshift-service-ca\") pod \"maas-api-e2e-mt-25c13375-7fb5867997-ggrbl\" (UID: \"c2078f8d-9360-43eb-9f01-7b690f07a53c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25c13375-7fb5867997-ggrbl"
Jul 28 22:40:42.893750 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:42.893726    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5952\" (UniqueName: \"kubernetes.io/projected/c2078f8d-9360-43eb-9f01-7b690f07a53c-kube-api-access-t5952\") pod \"maas-api-e2e-mt-25c13375-7fb5867997-ggrbl\" (UID: \"c2078f8d-9360-43eb-9f01-7b690f07a53c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25c13375-7fb5867997-ggrbl"
Jul 28 22:40:43.389247 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:43.389209    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2078f8d-9360-43eb-9f01-7b690f07a53c-maas-api-tls\") pod \"maas-api-e2e-mt-25c13375-7fb5867997-ggrbl\" (UID: \"c2078f8d-9360-43eb-9f01-7b690f07a53c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25c13375-7fb5867997-ggrbl"
Jul 28 22:40:43.391756 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:43.391728    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2078f8d-9360-43eb-9f01-7b690f07a53c-maas-api-tls\") pod \"maas-api-e2e-mt-25c13375-7fb5867997-ggrbl\" (UID: \"c2078f8d-9360-43eb-9f01-7b690f07a53c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25c13375-7fb5867997-ggrbl"
Jul 28 22:40:43.447297 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:43.447260    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25c13375-7cfb549784-4glbg"]
Jul 28 22:40:43.450778 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:43.450755    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-25c13375-7cfb549784-4glbg"
Jul 28 22:40:43.453060 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:43.453040    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-25c13375\""
Jul 28 22:40:43.453144 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:43.453067    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-25c13375-dockercfg-w5bmh\""
Jul 28 22:40:43.461790 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:43.461768    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25c13375-7cfb549784-4glbg"]
Jul 28 22:40:43.480390 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:43.480360    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25c13375-7d98d5c7f8-79f9p"]
Jul 28 22:40:43.483691 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:43.483671    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-25c13375-7d98d5c7f8-79f9p"
Jul 28 22:40:43.496244 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:43.496221    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25c13375-7d98d5c7f8-79f9p"]
Jul 28 22:40:43.563833 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:43.563786    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25c13375-7fb5867997-ggrbl"
Jul 28 22:40:43.590754 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:43.590725    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bctpb\" (UniqueName: \"kubernetes.io/projected/c0b4ad32-d026-457c-9859-26caaed5f1d6-kube-api-access-bctpb\") pod \"payload-processing-e2e-mt-25c13375-7d98d5c7f8-79f9p\" (UID: \"c0b4ad32-d026-457c-9859-26caaed5f1d6\") " pod="openshift-ingress/payload-processing-e2e-mt-25c13375-7d98d5c7f8-79f9p"
Jul 28 22:40:43.590909 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:43.590778    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6da2efc-4ed2-4b1d-9511-6e927950ddb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-25c13375-7cfb549784-4glbg\" (UID: \"d6da2efc-4ed2-4b1d-9511-6e927950ddb8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25c13375-7cfb549784-4glbg"
Jul 28 22:40:43.590909 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:43.590870    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr2hm\" (UniqueName: \"kubernetes.io/projected/d6da2efc-4ed2-4b1d-9511-6e927950ddb8-kube-api-access-fr2hm\") pod \"payload-pre-processing-e2e-mt-25c13375-7cfb549784-4glbg\" (UID: \"d6da2efc-4ed2-4b1d-9511-6e927950ddb8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25c13375-7cfb549784-4glbg"
Jul 28 22:40:43.590909 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:43.590889    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0b4ad32-d026-457c-9859-26caaed5f1d6-plugins-config-volume\") pod \"payload-processing-e2e-mt-25c13375-7d98d5c7f8-79f9p\" (UID: \"c0b4ad32-d026-457c-9859-26caaed5f1d6\") " pod="openshift-ingress/payload-processing-e2e-mt-25c13375-7d98d5c7f8-79f9p"
Jul 28 22:40:43.691956 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:43.691919    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr2hm\" (UniqueName: \"kubernetes.io/projected/d6da2efc-4ed2-4b1d-9511-6e927950ddb8-kube-api-access-fr2hm\") pod \"payload-pre-processing-e2e-mt-25c13375-7cfb549784-4glbg\" (UID: \"d6da2efc-4ed2-4b1d-9511-6e927950ddb8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25c13375-7cfb549784-4glbg"
Jul 28 22:40:43.692413 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:43.691960    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0b4ad32-d026-457c-9859-26caaed5f1d6-plugins-config-volume\") pod \"payload-processing-e2e-mt-25c13375-7d98d5c7f8-79f9p\" (UID: \"c0b4ad32-d026-457c-9859-26caaed5f1d6\") " pod="openshift-ingress/payload-processing-e2e-mt-25c13375-7d98d5c7f8-79f9p"
Jul 28 22:40:43.692413 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:43.691989    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bctpb\" (UniqueName: \"kubernetes.io/projected/c0b4ad32-d026-457c-9859-26caaed5f1d6-kube-api-access-bctpb\") pod \"payload-processing-e2e-mt-25c13375-7d98d5c7f8-79f9p\" (UID: \"c0b4ad32-d026-457c-9859-26caaed5f1d6\") " pod="openshift-ingress/payload-processing-e2e-mt-25c13375-7d98d5c7f8-79f9p"
Jul 28 22:40:43.692413 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:43.692017    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6da2efc-4ed2-4b1d-9511-6e927950ddb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-25c13375-7cfb549784-4glbg\" (UID: \"d6da2efc-4ed2-4b1d-9511-6e927950ddb8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25c13375-7cfb549784-4glbg"
Jul 28 22:40:43.693005 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:43.692979    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6da2efc-4ed2-4b1d-9511-6e927950ddb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-25c13375-7cfb549784-4glbg\" (UID: \"d6da2efc-4ed2-4b1d-9511-6e927950ddb8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25c13375-7cfb549784-4glbg"
Jul 28 22:40:43.693106 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:43.692988    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0b4ad32-d026-457c-9859-26caaed5f1d6-plugins-config-volume\") pod \"payload-processing-e2e-mt-25c13375-7d98d5c7f8-79f9p\" (UID: \"c0b4ad32-d026-457c-9859-26caaed5f1d6\") " pod="openshift-ingress/payload-processing-e2e-mt-25c13375-7d98d5c7f8-79f9p"
Jul 28 22:40:43.700396 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:43.700346    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bctpb\" (UniqueName: \"kubernetes.io/projected/c0b4ad32-d026-457c-9859-26caaed5f1d6-kube-api-access-bctpb\") pod \"payload-processing-e2e-mt-25c13375-7d98d5c7f8-79f9p\" (UID: \"c0b4ad32-d026-457c-9859-26caaed5f1d6\") " pod="openshift-ingress/payload-processing-e2e-mt-25c13375-7d98d5c7f8-79f9p"
Jul 28 22:40:43.700596 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:43.700577    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr2hm\" (UniqueName: \"kubernetes.io/projected/d6da2efc-4ed2-4b1d-9511-6e927950ddb8-kube-api-access-fr2hm\") pod \"payload-pre-processing-e2e-mt-25c13375-7cfb549784-4glbg\" (UID: \"d6da2efc-4ed2-4b1d-9511-6e927950ddb8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25c13375-7cfb549784-4glbg"
Jul 28 22:40:43.707191 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:43.707170    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25c13375-7fb5867997-ggrbl"]
Jul 28 22:40:43.709491 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:40:43.709467    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2078f8d_9360_43eb_9f01_7b690f07a53c.slice/crio-0f74482f103d1d11250f8f76312c6c9fa04449f4f58f2dd80e9634dcb58a6656 WatchSource:0}: Error finding container 0f74482f103d1d11250f8f76312c6c9fa04449f4f58f2dd80e9634dcb58a6656: Status 404 returned error can't find the container with id 0f74482f103d1d11250f8f76312c6c9fa04449f4f58f2dd80e9634dcb58a6656
Jul 28 22:40:43.710882 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:43.710864    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 22:40:43.762529 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:43.762502    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-25c13375-7cfb549784-4glbg"
Jul 28 22:40:43.793025 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:43.792993    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-25c13375-7d98d5c7f8-79f9p"
Jul 28 22:40:43.912173 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:43.912091    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25c13375-7cfb549784-4glbg"]
Jul 28 22:40:43.916306 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:40:43.916239    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6da2efc_4ed2_4b1d_9511_6e927950ddb8.slice/crio-0b4ec2bcaf29193ade85b34d1c43d244dcda9b39bbb3e2774f151be277c71646 WatchSource:0}: Error finding container 0b4ec2bcaf29193ade85b34d1c43d244dcda9b39bbb3e2774f151be277c71646: Status 404 returned error can't find the container with id 0b4ec2bcaf29193ade85b34d1c43d244dcda9b39bbb3e2774f151be277c71646
Jul 28 22:40:44.146758 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:44.146727    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25c13375-7d98d5c7f8-79f9p"]
Jul 28 22:40:44.148852 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:40:44.148819    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b4ad32_d026_457c_9859_26caaed5f1d6.slice/crio-01f9dea02839382469f8b9fe611f6ff59374cf698d75a2beee9674c2b58fe718 WatchSource:0}: Error finding container 01f9dea02839382469f8b9fe611f6ff59374cf698d75a2beee9674c2b58fe718: Status 404 returned error can't find the container with id 01f9dea02839382469f8b9fe611f6ff59374cf698d75a2beee9674c2b58fe718
Jul 28 22:40:44.467824 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:44.467773    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25c13375-7fb5867997-ggrbl" event={"ID":"c2078f8d-9360-43eb-9f01-7b690f07a53c","Type":"ContainerStarted","Data":"fc30dd44480f031593418b5ce3232ad14a7390dd67304370df0ea56a21141d0d"}
Jul 28 22:40:44.468030 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:44.467843    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25c13375-7fb5867997-ggrbl" event={"ID":"c2078f8d-9360-43eb-9f01-7b690f07a53c","Type":"ContainerStarted","Data":"0f74482f103d1d11250f8f76312c6c9fa04449f4f58f2dd80e9634dcb58a6656"}
Jul 28 22:40:44.468030 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:44.467876    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25c13375-7fb5867997-ggrbl"
Jul 28 22:40:44.469465 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:44.469434    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-25c13375-7d98d5c7f8-79f9p" event={"ID":"c0b4ad32-d026-457c-9859-26caaed5f1d6","Type":"ContainerStarted","Data":"b92e5432b2b4f5a583d24afdb5bc5312a1854483ca1ec89400998ed1cd0ae080"}
Jul 28 22:40:44.469589 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:44.469489    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-25c13375-7d98d5c7f8-79f9p" event={"ID":"c0b4ad32-d026-457c-9859-26caaed5f1d6","Type":"ContainerStarted","Data":"01f9dea02839382469f8b9fe611f6ff59374cf698d75a2beee9674c2b58fe718"}
Jul 28 22:40:44.469589 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:44.469543    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-25c13375-7d98d5c7f8-79f9p"
Jul 28 22:40:44.470972 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:44.470950    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-25c13375-7cfb549784-4glbg" event={"ID":"d6da2efc-4ed2-4b1d-9511-6e927950ddb8","Type":"ContainerStarted","Data":"242c68194fdd70e8b89aceb73b964daaaa062e94064777bcd5632280ef748672"}
Jul 28 22:40:44.471083 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:44.470985    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-25c13375-7cfb549784-4glbg" event={"ID":"d6da2efc-4ed2-4b1d-9511-6e927950ddb8","Type":"ContainerStarted","Data":"0b4ec2bcaf29193ade85b34d1c43d244dcda9b39bbb3e2774f151be277c71646"}
Jul 28 22:40:44.471136 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:44.471084    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-25c13375-7cfb549784-4glbg"
Jul 28 22:40:44.485602 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:44.485558    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25c13375-7fb5867997-ggrbl" podStartSLOduration=2.013077254 podStartE2EDuration="2.485545701s" totalImagesPullingTime="472.468447ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:40:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:40:44.483447491 +0000 UTC m=+2141.863585134" watchObservedRunningTime="2026-07-28 22:40:44.485545701 +0000 UTC m=+2141.865683382"
Jul 28 22:40:44.500238 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:44.500199    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-25c13375-7d98d5c7f8-79f9p" podStartSLOduration=1.5001882069999999 podStartE2EDuration="1.500188207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:40:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:40:44.498390624 +0000 UTC m=+2141.878528270" watchObservedRunningTime="2026-07-28 22:40:44.500188207 +0000 UTC m=+2141.880325877"
Jul 28 22:40:44.521466 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:44.521429    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-25c13375-7cfb549784-4glbg" podStartSLOduration=1.521413041 podStartE2EDuration="1.521413041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:40:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:40:44.520537732 +0000 UTC m=+2141.900675378" watchObservedRunningTime="2026-07-28 22:40:44.521413041 +0000 UTC m=+2141.901550688"
Jul 28 22:40:50.481522 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:50.481474    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25c13375-7fb5867997-ggrbl"
Jul 28 22:40:55.478587 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:55.478503    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-25c13375-7d98d5c7f8-79f9p"
Jul 28 22:40:55.478970 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:40:55.478692    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-25c13375-7cfb549784-4glbg"
Jul 28 22:41:57.517627 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:57.517593    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25c13375-7fb5867997-ggrbl"]
Jul 28 22:41:57.518111 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:57.517792    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25c13375-7fb5867997-ggrbl" podUID="c2078f8d-9360-43eb-9f01-7b690f07a53c" containerName="maas-api" containerID="cri-o://fc30dd44480f031593418b5ce3232ad14a7390dd67304370df0ea56a21141d0d" gracePeriod=30
Jul 28 22:41:57.562600 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:57.562567    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25c13375-7d98d5c7f8-79f9p"]
Jul 28 22:41:57.562845 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:57.562815    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-25c13375-7d98d5c7f8-79f9p" podUID="c0b4ad32-d026-457c-9859-26caaed5f1d6" containerName="payload-processing" containerID="cri-o://b92e5432b2b4f5a583d24afdb5bc5312a1854483ca1ec89400998ed1cd0ae080" gracePeriod=30
Jul 28 22:41:57.568506 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:57.568479    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25c13375-7cfb549784-4glbg"]
Jul 28 22:41:57.568809 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:57.568785    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-25c13375-7cfb549784-4glbg" podUID="d6da2efc-4ed2-4b1d-9511-6e927950ddb8" containerName="payload-pre-processing" containerID="cri-o://242c68194fdd70e8b89aceb73b964daaaa062e94064777bcd5632280ef748672" gracePeriod=30
Jul 28 22:41:57.825608 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:57.825573    2563 generic.go:356] "Generic (PLEG): container finished" podID="c0b4ad32-d026-457c-9859-26caaed5f1d6" containerID="b92e5432b2b4f5a583d24afdb5bc5312a1854483ca1ec89400998ed1cd0ae080" exitCode=0
Jul 28 22:41:57.825759 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:57.825652    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-25c13375-7d98d5c7f8-79f9p" event={"ID":"c0b4ad32-d026-457c-9859-26caaed5f1d6","Type":"ContainerDied","Data":"b92e5432b2b4f5a583d24afdb5bc5312a1854483ca1ec89400998ed1cd0ae080"}
Jul 28 22:41:57.825759 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:57.825724    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-25c13375-7d98d5c7f8-79f9p" event={"ID":"c0b4ad32-d026-457c-9859-26caaed5f1d6","Type":"ContainerDied","Data":"01f9dea02839382469f8b9fe611f6ff59374cf698d75a2beee9674c2b58fe718"}
Jul 28 22:41:57.825759 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:57.825751    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01f9dea02839382469f8b9fe611f6ff59374cf698d75a2beee9674c2b58fe718"
Jul 28 22:41:57.827784 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:57.827762    2563 generic.go:356] "Generic (PLEG): container finished" podID="d6da2efc-4ed2-4b1d-9511-6e927950ddb8" containerID="242c68194fdd70e8b89aceb73b964daaaa062e94064777bcd5632280ef748672" exitCode=0
Jul 28 22:41:57.827913 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:57.827838    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-25c13375-7cfb549784-4glbg" event={"ID":"d6da2efc-4ed2-4b1d-9511-6e927950ddb8","Type":"ContainerDied","Data":"242c68194fdd70e8b89aceb73b964daaaa062e94064777bcd5632280ef748672"}
Jul 28 22:41:57.829712 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:57.829688    2563 generic.go:356] "Generic (PLEG): container finished" podID="c2078f8d-9360-43eb-9f01-7b690f07a53c" containerID="fc30dd44480f031593418b5ce3232ad14a7390dd67304370df0ea56a21141d0d" exitCode=0
Jul 28 22:41:57.829813 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:57.829766    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25c13375-7fb5867997-ggrbl" event={"ID":"c2078f8d-9360-43eb-9f01-7b690f07a53c","Type":"ContainerDied","Data":"fc30dd44480f031593418b5ce3232ad14a7390dd67304370df0ea56a21141d0d"}
Jul 28 22:41:57.833151 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:57.833129    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-25c13375-7d98d5c7f8-79f9p"
Jul 28 22:41:57.869415 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:57.869392    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-25c13375-7cfb549784-4glbg"
Jul 28 22:41:57.872768 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:57.872749    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25c13375-7fb5867997-ggrbl"
Jul 28 22:41:57.947815 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:57.947774    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2078f8d-9360-43eb-9f01-7b690f07a53c-maas-api-tls\") pod \"c2078f8d-9360-43eb-9f01-7b690f07a53c\" (UID: \"c2078f8d-9360-43eb-9f01-7b690f07a53c\") "
Jul 28 22:41:57.947999 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:57.947832    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6da2efc-4ed2-4b1d-9511-6e927950ddb8-plugins-config-volume\") pod \"d6da2efc-4ed2-4b1d-9511-6e927950ddb8\" (UID: \"d6da2efc-4ed2-4b1d-9511-6e927950ddb8\") "
Jul 28 22:41:57.947999 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:57.947872    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5952\" (UniqueName: \"kubernetes.io/projected/c2078f8d-9360-43eb-9f01-7b690f07a53c-kube-api-access-t5952\") pod \"c2078f8d-9360-43eb-9f01-7b690f07a53c\" (UID: \"c2078f8d-9360-43eb-9f01-7b690f07a53c\") "
Jul 28 22:41:57.947999 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:57.947891    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2078f8d-9360-43eb-9f01-7b690f07a53c-openshift-service-ca\") pod \"c2078f8d-9360-43eb-9f01-7b690f07a53c\" (UID: \"c2078f8d-9360-43eb-9f01-7b690f07a53c\") "
Jul 28 22:41:57.947999 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:57.947917    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0b4ad32-d026-457c-9859-26caaed5f1d6-plugins-config-volume\") pod \"c0b4ad32-d026-457c-9859-26caaed5f1d6\" (UID: \"c0b4ad32-d026-457c-9859-26caaed5f1d6\") "
Jul 28 22:41:57.947999 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:57.947955    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bctpb\" (UniqueName: \"kubernetes.io/projected/c0b4ad32-d026-457c-9859-26caaed5f1d6-kube-api-access-bctpb\") pod \"c0b4ad32-d026-457c-9859-26caaed5f1d6\" (UID: \"c0b4ad32-d026-457c-9859-26caaed5f1d6\") "
Jul 28 22:41:57.948276 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:57.948003    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fr2hm\" (UniqueName: \"kubernetes.io/projected/d6da2efc-4ed2-4b1d-9511-6e927950ddb8-kube-api-access-fr2hm\") pod \"d6da2efc-4ed2-4b1d-9511-6e927950ddb8\" (UID: \"d6da2efc-4ed2-4b1d-9511-6e927950ddb8\") "
Jul 28 22:41:57.948344 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:57.948285    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2078f8d-9360-43eb-9f01-7b690f07a53c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c2078f8d-9360-43eb-9f01-7b690f07a53c" (UID: "c2078f8d-9360-43eb-9f01-7b690f07a53c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:41:57.948344 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:57.948314    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0b4ad32-d026-457c-9859-26caaed5f1d6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c0b4ad32-d026-457c-9859-26caaed5f1d6" (UID: "c0b4ad32-d026-457c-9859-26caaed5f1d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:41:57.948344 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:57.948330    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6da2efc-4ed2-4b1d-9511-6e927950ddb8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d6da2efc-4ed2-4b1d-9511-6e927950ddb8" (UID: "d6da2efc-4ed2-4b1d-9511-6e927950ddb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:41:57.950177 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:57.950150    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2078f8d-9360-43eb-9f01-7b690f07a53c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c2078f8d-9360-43eb-9f01-7b690f07a53c" (UID: "c2078f8d-9360-43eb-9f01-7b690f07a53c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:41:57.950177 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:57.950167    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2078f8d-9360-43eb-9f01-7b690f07a53c-kube-api-access-t5952" (OuterVolumeSpecName: "kube-api-access-t5952") pod "c2078f8d-9360-43eb-9f01-7b690f07a53c" (UID: "c2078f8d-9360-43eb-9f01-7b690f07a53c"). InnerVolumeSpecName "kube-api-access-t5952". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:41:57.950331 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:57.950195    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6da2efc-4ed2-4b1d-9511-6e927950ddb8-kube-api-access-fr2hm" (OuterVolumeSpecName: "kube-api-access-fr2hm") pod "d6da2efc-4ed2-4b1d-9511-6e927950ddb8" (UID: "d6da2efc-4ed2-4b1d-9511-6e927950ddb8"). InnerVolumeSpecName "kube-api-access-fr2hm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:41:57.950331 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:57.950207    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0b4ad32-d026-457c-9859-26caaed5f1d6-kube-api-access-bctpb" (OuterVolumeSpecName: "kube-api-access-bctpb") pod "c0b4ad32-d026-457c-9859-26caaed5f1d6" (UID: "c0b4ad32-d026-457c-9859-26caaed5f1d6"). InnerVolumeSpecName "kube-api-access-bctpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:41:58.049330 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:58.049241    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bctpb\" (UniqueName: \"kubernetes.io/projected/c0b4ad32-d026-457c-9859-26caaed5f1d6-kube-api-access-bctpb\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:41:58.049330 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:58.049269    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr2hm\" (UniqueName: \"kubernetes.io/projected/d6da2efc-4ed2-4b1d-9511-6e927950ddb8-kube-api-access-fr2hm\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:41:58.049330 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:58.049279    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2078f8d-9360-43eb-9f01-7b690f07a53c-maas-api-tls\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:41:58.049330 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:58.049298    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6da2efc-4ed2-4b1d-9511-6e927950ddb8-plugins-config-volume\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:41:58.049330 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:58.049307    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5952\" (UniqueName: \"kubernetes.io/projected/c2078f8d-9360-43eb-9f01-7b690f07a53c-kube-api-access-t5952\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:41:58.049330 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:58.049315    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2078f8d-9360-43eb-9f01-7b690f07a53c-openshift-service-ca\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:41:58.049330 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:58.049325    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0b4ad32-d026-457c-9859-26caaed5f1d6-plugins-config-volume\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:41:58.835734 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:58.835636    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-25c13375-7cfb549784-4glbg" event={"ID":"d6da2efc-4ed2-4b1d-9511-6e927950ddb8","Type":"ContainerDied","Data":"0b4ec2bcaf29193ade85b34d1c43d244dcda9b39bbb3e2774f151be277c71646"}
Jul 28 22:41:58.835734 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:58.835666    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-25c13375-7cfb549784-4glbg"
Jul 28 22:41:58.836233 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:58.835712    2563 scope.go:117] "RemoveContainer" containerID="242c68194fdd70e8b89aceb73b964daaaa062e94064777bcd5632280ef748672"
Jul 28 22:41:58.838022 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:58.837995    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-25c13375-7d98d5c7f8-79f9p"
Jul 28 22:41:58.848137 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:58.838586    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25c13375-7fb5867997-ggrbl" event={"ID":"c2078f8d-9360-43eb-9f01-7b690f07a53c","Type":"ContainerDied","Data":"0f74482f103d1d11250f8f76312c6c9fa04449f4f58f2dd80e9634dcb58a6656"}
Jul 28 22:41:58.848137 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:58.840734    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25c13375-7fb5867997-ggrbl"
Jul 28 22:41:58.852865 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:58.852846    2563 scope.go:117] "RemoveContainer" containerID="fc30dd44480f031593418b5ce3232ad14a7390dd67304370df0ea56a21141d0d"
Jul 28 22:41:58.870422 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:58.870399    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25c13375-7cfb549784-4glbg"]
Jul 28 22:41:58.874675 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:58.874655    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25c13375-7cfb549784-4glbg"]
Jul 28 22:41:58.884050 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:58.884030    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25c13375-7fb5867997-ggrbl"]
Jul 28 22:41:58.890251 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:58.890232    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25c13375-7fb5867997-ggrbl"]
Jul 28 22:41:58.900343 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:58.900318    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25c13375-7d98d5c7f8-79f9p"]
Jul 28 22:41:58.903544 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:58.903523    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25c13375-7d98d5c7f8-79f9p"]
Jul 28 22:41:59.130252 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:59.130173    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0b4ad32-d026-457c-9859-26caaed5f1d6" path="/var/lib/kubelet/pods/c0b4ad32-d026-457c-9859-26caaed5f1d6/volumes"
Jul 28 22:41:59.130625 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:59.130610    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2078f8d-9360-43eb-9f01-7b690f07a53c" path="/var/lib/kubelet/pods/c2078f8d-9360-43eb-9f01-7b690f07a53c/volumes"
Jul 28 22:41:59.130922 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:41:59.130911    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6da2efc-4ed2-4b1d-9511-6e927950ddb8" path="/var/lib/kubelet/pods/d6da2efc-4ed2-4b1d-9511-6e927950ddb8/volumes"
Jul 28 22:44:01.618407 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:01.618307    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9216b7-547d99479c-bn7dm"]
Jul 28 22:44:01.618946 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:01.618819    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6da2efc-4ed2-4b1d-9511-6e927950ddb8" containerName="payload-pre-processing"
Jul 28 22:44:01.618946 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:01.618839    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6da2efc-4ed2-4b1d-9511-6e927950ddb8" containerName="payload-pre-processing"
Jul 28 22:44:01.618946 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:01.618851    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2078f8d-9360-43eb-9f01-7b690f07a53c" containerName="maas-api"
Jul 28 22:44:01.618946 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:01.618861    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2078f8d-9360-43eb-9f01-7b690f07a53c" containerName="maas-api"
Jul 28 22:44:01.618946 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:01.618877    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0b4ad32-d026-457c-9859-26caaed5f1d6" containerName="payload-processing"
Jul 28 22:44:01.618946 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:01.618886    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0b4ad32-d026-457c-9859-26caaed5f1d6" containerName="payload-processing"
Jul 28 22:44:01.619246 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:01.618986    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2078f8d-9360-43eb-9f01-7b690f07a53c" containerName="maas-api"
Jul 28 22:44:01.619246 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:01.618999    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0b4ad32-d026-457c-9859-26caaed5f1d6" containerName="payload-processing"
Jul 28 22:44:01.619246 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:01.619012    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6da2efc-4ed2-4b1d-9511-6e927950ddb8" containerName="payload-pre-processing"
Jul 28 22:44:01.622273 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:01.622254    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9216b7-547d99479c-bn7dm"
Jul 28 22:44:01.624754 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:01.624736    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-9216b7-serving-cert\""
Jul 28 22:44:01.633047 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:01.633026    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9216b7-547d99479c-bn7dm"]
Jul 28 22:44:01.705104 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:01.705064    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/377c437e-7fd1-43ba-8a26-6450f73d708c-maas-api-tls\") pod \"maas-api-e2e-inf-a-9216b7-547d99479c-bn7dm\" (UID: \"377c437e-7fd1-43ba-8a26-6450f73d708c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9216b7-547d99479c-bn7dm"
Jul 28 22:44:01.705278 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:01.705137    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/377c437e-7fd1-43ba-8a26-6450f73d708c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9216b7-547d99479c-bn7dm\" (UID: \"377c437e-7fd1-43ba-8a26-6450f73d708c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9216b7-547d99479c-bn7dm"
Jul 28 22:44:01.705278 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:01.705167    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2hfx\" (UniqueName: \"kubernetes.io/projected/377c437e-7fd1-43ba-8a26-6450f73d708c-kube-api-access-v2hfx\") pod \"maas-api-e2e-inf-a-9216b7-547d99479c-bn7dm\" (UID: \"377c437e-7fd1-43ba-8a26-6450f73d708c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9216b7-547d99479c-bn7dm"
Jul 28 22:44:01.806468 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:01.806426    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/377c437e-7fd1-43ba-8a26-6450f73d708c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9216b7-547d99479c-bn7dm\" (UID: \"377c437e-7fd1-43ba-8a26-6450f73d708c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9216b7-547d99479c-bn7dm"
Jul 28 22:44:01.806677 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:01.806483    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2hfx\" (UniqueName: \"kubernetes.io/projected/377c437e-7fd1-43ba-8a26-6450f73d708c-kube-api-access-v2hfx\") pod \"maas-api-e2e-inf-a-9216b7-547d99479c-bn7dm\" (UID: \"377c437e-7fd1-43ba-8a26-6450f73d708c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9216b7-547d99479c-bn7dm"
Jul 28 22:44:01.806677 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:01.806548    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/377c437e-7fd1-43ba-8a26-6450f73d708c-maas-api-tls\") pod \"maas-api-e2e-inf-a-9216b7-547d99479c-bn7dm\" (UID: \"377c437e-7fd1-43ba-8a26-6450f73d708c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9216b7-547d99479c-bn7dm"
Jul 28 22:44:01.806810 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:44:01.806693    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-9216b7-serving-cert: secret "maas-api-e2e-inf-a-9216b7-serving-cert" not found
Jul 28 22:44:01.806810 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:44:01.806774    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/377c437e-7fd1-43ba-8a26-6450f73d708c-maas-api-tls podName:377c437e-7fd1-43ba-8a26-6450f73d708c nodeName:}" failed. No retries permitted until 2026-07-28 22:44:02.306753643 +0000 UTC m=+2339.686891280 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/377c437e-7fd1-43ba-8a26-6450f73d708c-maas-api-tls") pod "maas-api-e2e-inf-a-9216b7-547d99479c-bn7dm" (UID: "377c437e-7fd1-43ba-8a26-6450f73d708c") : secret "maas-api-e2e-inf-a-9216b7-serving-cert" not found
Jul 28 22:44:01.807175 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:01.807154    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/377c437e-7fd1-43ba-8a26-6450f73d708c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9216b7-547d99479c-bn7dm\" (UID: \"377c437e-7fd1-43ba-8a26-6450f73d708c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9216b7-547d99479c-bn7dm"
Jul 28 22:44:01.816988 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:01.816960    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2hfx\" (UniqueName: \"kubernetes.io/projected/377c437e-7fd1-43ba-8a26-6450f73d708c-kube-api-access-v2hfx\") pod \"maas-api-e2e-inf-a-9216b7-547d99479c-bn7dm\" (UID: \"377c437e-7fd1-43ba-8a26-6450f73d708c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9216b7-547d99479c-bn7dm"
Jul 28 22:44:02.311843 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:02.311805    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/377c437e-7fd1-43ba-8a26-6450f73d708c-maas-api-tls\") pod \"maas-api-e2e-inf-a-9216b7-547d99479c-bn7dm\" (UID: \"377c437e-7fd1-43ba-8a26-6450f73d708c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9216b7-547d99479c-bn7dm"
Jul 28 22:44:02.314465 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:02.314443    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/377c437e-7fd1-43ba-8a26-6450f73d708c-maas-api-tls\") pod \"maas-api-e2e-inf-a-9216b7-547d99479c-bn7dm\" (UID: \"377c437e-7fd1-43ba-8a26-6450f73d708c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9216b7-547d99479c-bn7dm"
Jul 28 22:44:02.403825 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:02.403792    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9216b7-76866769cb-7c2sl"]
Jul 28 22:44:02.407170 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:02.407152    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9216b7-76866769cb-7c2sl"
Jul 28 22:44:02.409756 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:02.409731    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-9216b7\""
Jul 28 22:44:02.409894 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:02.409734    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-9216b7-dockercfg-hr87m\""
Jul 28 22:44:02.417964 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:02.417940    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9216b7-76866769cb-7c2sl"]
Jul 28 22:44:02.453614 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:02.453579    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9216b7-6d8f8c4cd8-vzbhn"]
Jul 28 22:44:02.456848 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:02.456827    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9216b7-6d8f8c4cd8-vzbhn"
Jul 28 22:44:02.468961 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:02.468933    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9216b7-6d8f8c4cd8-vzbhn"]
Jul 28 22:44:02.513634 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:02.513605    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5lrt\" (UniqueName: \"kubernetes.io/projected/6efb5282-8f18-4e60-aa94-287529eeeea4-kube-api-access-g5lrt\") pod \"payload-pre-processing-e2e-inf-a-9216b7-76866769cb-7c2sl\" (UID: \"6efb5282-8f18-4e60-aa94-287529eeeea4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9216b7-76866769cb-7c2sl"
Jul 28 22:44:02.513811 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:02.513656    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6efb5282-8f18-4e60-aa94-287529eeeea4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9216b7-76866769cb-7c2sl\" (UID: \"6efb5282-8f18-4e60-aa94-287529eeeea4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9216b7-76866769cb-7c2sl"
Jul 28 22:44:02.533551 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:02.533525    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9216b7-547d99479c-bn7dm"
Jul 28 22:44:02.614535 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:02.614505    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5lrt\" (UniqueName: \"kubernetes.io/projected/6efb5282-8f18-4e60-aa94-287529eeeea4-kube-api-access-g5lrt\") pod \"payload-pre-processing-e2e-inf-a-9216b7-76866769cb-7c2sl\" (UID: \"6efb5282-8f18-4e60-aa94-287529eeeea4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9216b7-76866769cb-7c2sl"
Jul 28 22:44:02.614706 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:02.614542    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b05f2b3d-98f9-4b02-9df7-3952783fb6e8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9216b7-6d8f8c4cd8-vzbhn\" (UID: \"b05f2b3d-98f9-4b02-9df7-3952783fb6e8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9216b7-6d8f8c4cd8-vzbhn"
Jul 28 22:44:02.614706 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:02.614569    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mq9f\" (UniqueName: \"kubernetes.io/projected/b05f2b3d-98f9-4b02-9df7-3952783fb6e8-kube-api-access-8mq9f\") pod \"payload-processing-e2e-inf-a-9216b7-6d8f8c4cd8-vzbhn\" (UID: \"b05f2b3d-98f9-4b02-9df7-3952783fb6e8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9216b7-6d8f8c4cd8-vzbhn"
Jul 28 22:44:02.614706 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:02.614683    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6efb5282-8f18-4e60-aa94-287529eeeea4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9216b7-76866769cb-7c2sl\" (UID: \"6efb5282-8f18-4e60-aa94-287529eeeea4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9216b7-76866769cb-7c2sl"
Jul 28 22:44:02.615472 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:02.615450    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6efb5282-8f18-4e60-aa94-287529eeeea4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9216b7-76866769cb-7c2sl\" (UID: \"6efb5282-8f18-4e60-aa94-287529eeeea4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9216b7-76866769cb-7c2sl"
Jul 28 22:44:02.626722 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:02.626697    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5lrt\" (UniqueName: \"kubernetes.io/projected/6efb5282-8f18-4e60-aa94-287529eeeea4-kube-api-access-g5lrt\") pod \"payload-pre-processing-e2e-inf-a-9216b7-76866769cb-7c2sl\" (UID: \"6efb5282-8f18-4e60-aa94-287529eeeea4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9216b7-76866769cb-7c2sl"
Jul 28 22:44:02.680453 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:02.680424    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9216b7-547d99479c-bn7dm"]
Jul 28 22:44:02.682842 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:44:02.682812    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod377c437e_7fd1_43ba_8a26_6450f73d708c.slice/crio-fa8b7752c5c0e5345e1c515874c676d38965443efd9354fc18fa98b26db81512 WatchSource:0}: Error finding container fa8b7752c5c0e5345e1c515874c676d38965443efd9354fc18fa98b26db81512: Status 404 returned error can't find the container with id fa8b7752c5c0e5345e1c515874c676d38965443efd9354fc18fa98b26db81512
Jul 28 22:44:02.716099 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:02.716072    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b05f2b3d-98f9-4b02-9df7-3952783fb6e8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9216b7-6d8f8c4cd8-vzbhn\" (UID: \"b05f2b3d-98f9-4b02-9df7-3952783fb6e8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9216b7-6d8f8c4cd8-vzbhn"
Jul 28 22:44:02.716208 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:02.716109    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mq9f\" (UniqueName: \"kubernetes.io/projected/b05f2b3d-98f9-4b02-9df7-3952783fb6e8-kube-api-access-8mq9f\") pod \"payload-processing-e2e-inf-a-9216b7-6d8f8c4cd8-vzbhn\" (UID: \"b05f2b3d-98f9-4b02-9df7-3952783fb6e8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9216b7-6d8f8c4cd8-vzbhn"
Jul 28 22:44:02.716742 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:02.716723    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b05f2b3d-98f9-4b02-9df7-3952783fb6e8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9216b7-6d8f8c4cd8-vzbhn\" (UID: \"b05f2b3d-98f9-4b02-9df7-3952783fb6e8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9216b7-6d8f8c4cd8-vzbhn"
Jul 28 22:44:02.717989 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:02.717972    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9216b7-76866769cb-7c2sl"
Jul 28 22:44:02.724083 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:02.724063    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mq9f\" (UniqueName: \"kubernetes.io/projected/b05f2b3d-98f9-4b02-9df7-3952783fb6e8-kube-api-access-8mq9f\") pod \"payload-processing-e2e-inf-a-9216b7-6d8f8c4cd8-vzbhn\" (UID: \"b05f2b3d-98f9-4b02-9df7-3952783fb6e8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9216b7-6d8f8c4cd8-vzbhn"
Jul 28 22:44:02.771188 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:02.771154    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9216b7-6d8f8c4cd8-vzbhn"
Jul 28 22:44:02.876557 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:02.876524    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9216b7-76866769cb-7c2sl"]
Jul 28 22:44:02.878620 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:44:02.878584    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6efb5282_8f18_4e60_aa94_287529eeeea4.slice/crio-f4990fbf82af459fad8a4534ca44e6e171986fcefdd19131da61418550c161dd WatchSource:0}: Error finding container f4990fbf82af459fad8a4534ca44e6e171986fcefdd19131da61418550c161dd: Status 404 returned error can't find the container with id f4990fbf82af459fad8a4534ca44e6e171986fcefdd19131da61418550c161dd
Jul 28 22:44:02.927714 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:02.927688    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9216b7-6d8f8c4cd8-vzbhn"]
Jul 28 22:44:02.929875 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:44:02.929851    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb05f2b3d_98f9_4b02_9df7_3952783fb6e8.slice/crio-773338b9f45e9ffb85b0aff56107a95e3c83bd1aaa5e9fb2556f57a01a7a736b WatchSource:0}: Error finding container 773338b9f45e9ffb85b0aff56107a95e3c83bd1aaa5e9fb2556f57a01a7a736b: Status 404 returned error can't find the container with id 773338b9f45e9ffb85b0aff56107a95e3c83bd1aaa5e9fb2556f57a01a7a736b
Jul 28 22:44:03.408227 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:03.408118    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9216b7-547d99479c-bn7dm" event={"ID":"377c437e-7fd1-43ba-8a26-6450f73d708c","Type":"ContainerStarted","Data":"53d7c94f76b860db207d19c8c5c1a28c6c072961931a4ac7579d77cb2b02cae2"}
Jul 28 22:44:03.408417 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:03.408313    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9216b7-547d99479c-bn7dm"
Jul 28 22:44:03.408417 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:03.408374    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9216b7-547d99479c-bn7dm" event={"ID":"377c437e-7fd1-43ba-8a26-6450f73d708c","Type":"ContainerStarted","Data":"fa8b7752c5c0e5345e1c515874c676d38965443efd9354fc18fa98b26db81512"}
Jul 28 22:44:03.409709 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:03.409684    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9216b7-6d8f8c4cd8-vzbhn" event={"ID":"b05f2b3d-98f9-4b02-9df7-3952783fb6e8","Type":"ContainerStarted","Data":"5bc1016113c8dd133738e1b7984a080e3f3b9fb9c6a135f4bb4d9c71cd46fa8b"}
Jul 28 22:44:03.409838 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:03.409723    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9216b7-6d8f8c4cd8-vzbhn" event={"ID":"b05f2b3d-98f9-4b02-9df7-3952783fb6e8","Type":"ContainerStarted","Data":"773338b9f45e9ffb85b0aff56107a95e3c83bd1aaa5e9fb2556f57a01a7a736b"}
Jul 28 22:44:03.409838 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:03.409781    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9216b7-6d8f8c4cd8-vzbhn"
Jul 28 22:44:03.411119 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:03.411098    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9216b7-76866769cb-7c2sl" event={"ID":"6efb5282-8f18-4e60-aa94-287529eeeea4","Type":"ContainerStarted","Data":"35edc575bf4d868161edcb455b713346362e606fcfdbc4347e83cb5358d2257a"}
Jul 28 22:44:03.411200 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:03.411134    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9216b7-76866769cb-7c2sl" event={"ID":"6efb5282-8f18-4e60-aa94-287529eeeea4","Type":"ContainerStarted","Data":"f4990fbf82af459fad8a4534ca44e6e171986fcefdd19131da61418550c161dd"}
Jul 28 22:44:03.411270 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:03.411197    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9216b7-76866769cb-7c2sl"
Jul 28 22:44:03.425303 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:03.425258    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9216b7-547d99479c-bn7dm" podStartSLOduration=2.00679827 podStartE2EDuration="2.425243791s" totalImagesPullingTime="418.445521ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:44:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:44:03.424016329 +0000 UTC m=+2340.804153992" watchObservedRunningTime="2026-07-28 22:44:03.425243791 +0000 UTC m=+2340.805381476"
Jul 28 22:44:03.438941 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:03.438887    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9216b7-76866769cb-7c2sl" podStartSLOduration=1.438870184 podStartE2EDuration="1.438870184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:44:03.43765592 +0000 UTC m=+2340.817793565" watchObservedRunningTime="2026-07-28 22:44:03.438870184 +0000 UTC m=+2340.819007829"
Jul 28 22:44:03.453319 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:03.453271    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-9216b7-6d8f8c4cd8-vzbhn" podStartSLOduration=1.453258176 podStartE2EDuration="1.453258176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:44:03.452051844 +0000 UTC m=+2340.832189489" watchObservedRunningTime="2026-07-28 22:44:03.453258176 +0000 UTC m=+2340.833395819"
Jul 28 22:44:09.421934 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:09.421898    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9216b7-547d99479c-bn7dm"
Jul 28 22:44:14.418091 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:14.418045    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9216b7-76866769cb-7c2sl" podUID="6efb5282-8f18-4e60-aa94-287529eeeea4" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.63:9004: connect: connection refused"
Jul 28 22:44:14.418581 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:14.418506    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9216b7-6d8f8c4cd8-vzbhn"
Jul 28 22:44:24.417284 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:24.417252    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9216b7-76866769cb-7c2sl"
Jul 28 22:44:29.894035 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:29.894001    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dd9c88-558f6b84d5-rc5hv"]
Jul 28 22:44:29.897569 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:29.897547    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd9c88-558f6b84d5-rc5hv"
Jul 28 22:44:29.900086 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:29.900064    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-dd9c88-serving-cert\""
Jul 28 22:44:29.910527 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:29.910501    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dd9c88-558f6b84d5-rc5hv"]
Jul 28 22:44:29.956739 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:29.956705    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a9226b1-f3c5-4fc9-a127-c5e2c6708725-maas-api-tls\") pod \"maas-api-e2e-inf-b-dd9c88-558f6b84d5-rc5hv\" (UID: \"0a9226b1-f3c5-4fc9-a127-c5e2c6708725\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd9c88-558f6b84d5-rc5hv"
Jul 28 22:44:29.956892 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:29.956747    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n8jx\" (UniqueName: \"kubernetes.io/projected/0a9226b1-f3c5-4fc9-a127-c5e2c6708725-kube-api-access-6n8jx\") pod \"maas-api-e2e-inf-b-dd9c88-558f6b84d5-rc5hv\" (UID: \"0a9226b1-f3c5-4fc9-a127-c5e2c6708725\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd9c88-558f6b84d5-rc5hv"
Jul 28 22:44:29.956892 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:29.956826    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9226b1-f3c5-4fc9-a127-c5e2c6708725-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dd9c88-558f6b84d5-rc5hv\" (UID: \"0a9226b1-f3c5-4fc9-a127-c5e2c6708725\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd9c88-558f6b84d5-rc5hv"
Jul 28 22:44:30.058010 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:30.057975    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a9226b1-f3c5-4fc9-a127-c5e2c6708725-maas-api-tls\") pod \"maas-api-e2e-inf-b-dd9c88-558f6b84d5-rc5hv\" (UID: \"0a9226b1-f3c5-4fc9-a127-c5e2c6708725\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd9c88-558f6b84d5-rc5hv"
Jul 28 22:44:30.058010 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:30.058014    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n8jx\" (UniqueName: \"kubernetes.io/projected/0a9226b1-f3c5-4fc9-a127-c5e2c6708725-kube-api-access-6n8jx\") pod \"maas-api-e2e-inf-b-dd9c88-558f6b84d5-rc5hv\" (UID: \"0a9226b1-f3c5-4fc9-a127-c5e2c6708725\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd9c88-558f6b84d5-rc5hv"
Jul 28 22:44:30.058221 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:30.058058    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9226b1-f3c5-4fc9-a127-c5e2c6708725-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dd9c88-558f6b84d5-rc5hv\" (UID: \"0a9226b1-f3c5-4fc9-a127-c5e2c6708725\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd9c88-558f6b84d5-rc5hv"
Jul 28 22:44:30.058221 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:44:30.058125    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-dd9c88-serving-cert: secret "maas-api-e2e-inf-b-dd9c88-serving-cert" not found
Jul 28 22:44:30.058221 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:44:30.058190    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a9226b1-f3c5-4fc9-a127-c5e2c6708725-maas-api-tls podName:0a9226b1-f3c5-4fc9-a127-c5e2c6708725 nodeName:}" failed. No retries permitted until 2026-07-28 22:44:30.558173661 +0000 UTC m=+2367.938311284 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0a9226b1-f3c5-4fc9-a127-c5e2c6708725-maas-api-tls") pod "maas-api-e2e-inf-b-dd9c88-558f6b84d5-rc5hv" (UID: "0a9226b1-f3c5-4fc9-a127-c5e2c6708725") : secret "maas-api-e2e-inf-b-dd9c88-serving-cert" not found
Jul 28 22:44:30.058688 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:30.058670    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9226b1-f3c5-4fc9-a127-c5e2c6708725-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dd9c88-558f6b84d5-rc5hv\" (UID: \"0a9226b1-f3c5-4fc9-a127-c5e2c6708725\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd9c88-558f6b84d5-rc5hv"
Jul 28 22:44:30.066828 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:30.066804    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n8jx\" (UniqueName: \"kubernetes.io/projected/0a9226b1-f3c5-4fc9-a127-c5e2c6708725-kube-api-access-6n8jx\") pod \"maas-api-e2e-inf-b-dd9c88-558f6b84d5-rc5hv\" (UID: \"0a9226b1-f3c5-4fc9-a127-c5e2c6708725\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd9c88-558f6b84d5-rc5hv"
Jul 28 22:44:30.561965 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:30.561928    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a9226b1-f3c5-4fc9-a127-c5e2c6708725-maas-api-tls\") pod \"maas-api-e2e-inf-b-dd9c88-558f6b84d5-rc5hv\" (UID: \"0a9226b1-f3c5-4fc9-a127-c5e2c6708725\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd9c88-558f6b84d5-rc5hv"
Jul 28 22:44:30.564774 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:30.564745    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a9226b1-f3c5-4fc9-a127-c5e2c6708725-maas-api-tls\") pod \"maas-api-e2e-inf-b-dd9c88-558f6b84d5-rc5hv\" (UID: \"0a9226b1-f3c5-4fc9-a127-c5e2c6708725\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd9c88-558f6b84d5-rc5hv"
Jul 28 22:44:30.578556 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:30.578528    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dd9c88-5c87c5484d-fgx7f"]
Jul 28 22:44:30.582120 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:30.582099    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-dd9c88-5c87c5484d-fgx7f"
Jul 28 22:44:30.585266 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:30.585235    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-dd9c88\""
Jul 28 22:44:30.585419 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:30.585268    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-dd9c88-dockercfg-f97rg\""
Jul 28 22:44:30.592831 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:30.592805    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dd9c88-5c87c5484d-fgx7f"]
Jul 28 22:44:30.662885 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:30.662850    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/658ba086-c4f5-4911-af7a-1ac6bc17a47c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-dd9c88-5c87c5484d-fgx7f\" (UID: \"658ba086-c4f5-4911-af7a-1ac6bc17a47c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dd9c88-5c87c5484d-fgx7f"
Jul 28 22:44:30.663052 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:30.662944    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6rt5\" (UniqueName: \"kubernetes.io/projected/658ba086-c4f5-4911-af7a-1ac6bc17a47c-kube-api-access-b6rt5\") pod \"payload-processing-e2e-inf-b-dd9c88-5c87c5484d-fgx7f\" (UID: \"658ba086-c4f5-4911-af7a-1ac6bc17a47c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dd9c88-5c87c5484d-fgx7f"
Jul 28 22:44:30.764168 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:30.764139    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6rt5\" (UniqueName: \"kubernetes.io/projected/658ba086-c4f5-4911-af7a-1ac6bc17a47c-kube-api-access-b6rt5\") pod \"payload-processing-e2e-inf-b-dd9c88-5c87c5484d-fgx7f\" (UID: \"658ba086-c4f5-4911-af7a-1ac6bc17a47c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dd9c88-5c87c5484d-fgx7f"
Jul 28 22:44:30.764340 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:30.764183    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/658ba086-c4f5-4911-af7a-1ac6bc17a47c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-dd9c88-5c87c5484d-fgx7f\" (UID: \"658ba086-c4f5-4911-af7a-1ac6bc17a47c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dd9c88-5c87c5484d-fgx7f"
Jul 28 22:44:30.764863 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:30.764843    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/658ba086-c4f5-4911-af7a-1ac6bc17a47c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-dd9c88-5c87c5484d-fgx7f\" (UID: \"658ba086-c4f5-4911-af7a-1ac6bc17a47c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dd9c88-5c87c5484d-fgx7f"
Jul 28 22:44:30.774716 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:30.774687    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6rt5\" (UniqueName: \"kubernetes.io/projected/658ba086-c4f5-4911-af7a-1ac6bc17a47c-kube-api-access-b6rt5\") pod \"payload-processing-e2e-inf-b-dd9c88-5c87c5484d-fgx7f\" (UID: \"658ba086-c4f5-4911-af7a-1ac6bc17a47c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dd9c88-5c87c5484d-fgx7f"
Jul 28 22:44:30.810038 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:30.810005    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd9c88-558f6b84d5-rc5hv"
Jul 28 22:44:30.895956 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:30.895887    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-dd9c88-5c87c5484d-fgx7f"
Jul 28 22:44:30.950324 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:30.950298    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dd9c88-558f6b84d5-rc5hv"]
Jul 28 22:44:30.953264 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:44:30.953204    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a9226b1_f3c5_4fc9_a127_c5e2c6708725.slice/crio-f140699a0e9a5662559f295a1f6d8fec511da3dcd8c17d588b604f20b2959396 WatchSource:0}: Error finding container f140699a0e9a5662559f295a1f6d8fec511da3dcd8c17d588b604f20b2959396: Status 404 returned error can't find the container with id f140699a0e9a5662559f295a1f6d8fec511da3dcd8c17d588b604f20b2959396
Jul 28 22:44:31.041035 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:31.040999    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dd9c88-5c87c5484d-fgx7f"]
Jul 28 22:44:31.043678 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:44:31.043651    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod658ba086_c4f5_4911_af7a_1ac6bc17a47c.slice/crio-4fb34edce2a527c8f5ec468d9a46a5b96af6833bced3eb5506175f3b91217360 WatchSource:0}: Error finding container 4fb34edce2a527c8f5ec468d9a46a5b96af6833bced3eb5506175f3b91217360: Status 404 returned error can't find the container with id 4fb34edce2a527c8f5ec468d9a46a5b96af6833bced3eb5506175f3b91217360
Jul 28 22:44:31.556342 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:31.556306    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd9c88-558f6b84d5-rc5hv" event={"ID":"0a9226b1-f3c5-4fc9-a127-c5e2c6708725","Type":"ContainerStarted","Data":"e8f83e71f268af35a10078dff5a19df1ac43bfe64ef09a2c1130cab5cd24aac2"}
Jul 28 22:44:31.556532 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:31.556394    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd9c88-558f6b84d5-rc5hv" event={"ID":"0a9226b1-f3c5-4fc9-a127-c5e2c6708725","Type":"ContainerStarted","Data":"f140699a0e9a5662559f295a1f6d8fec511da3dcd8c17d588b604f20b2959396"}
Jul 28 22:44:31.556632 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:31.556530    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd9c88-558f6b84d5-rc5hv"
Jul 28 22:44:31.558111 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:31.558085    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-dd9c88-5c87c5484d-fgx7f" event={"ID":"658ba086-c4f5-4911-af7a-1ac6bc17a47c","Type":"ContainerStarted","Data":"91d4c13b73f184dbe05ef4491047ab2332301a915079614faa3c7cba153e30de"}
Jul 28 22:44:31.558235 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:31.558130    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-dd9c88-5c87c5484d-fgx7f" event={"ID":"658ba086-c4f5-4911-af7a-1ac6bc17a47c","Type":"ContainerStarted","Data":"4fb34edce2a527c8f5ec468d9a46a5b96af6833bced3eb5506175f3b91217360"}
Jul 28 22:44:31.558235 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:31.558159    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-dd9c88-5c87c5484d-fgx7f"
Jul 28 22:44:31.574199 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:31.574153    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd9c88-558f6b84d5-rc5hv" podStartSLOduration=2.228844762 podStartE2EDuration="2.574138719s" totalImagesPullingTime="345.293957ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:44:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:44:31.573323341 +0000 UTC m=+2368.953460983" watchObservedRunningTime="2026-07-28 22:44:31.574138719 +0000 UTC m=+2368.954276363"
Jul 28 22:44:31.593586 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:31.593488    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-dd9c88-5c87c5484d-fgx7f" podStartSLOduration=1.593467381 podStartE2EDuration="1.593467381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:44:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:44:31.591392207 +0000 UTC m=+2368.971529851" watchObservedRunningTime="2026-07-28 22:44:31.593467381 +0000 UTC m=+2368.973605025"
Jul 28 22:44:37.568966 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:37.568937    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd9c88-558f6b84d5-rc5hv"
Jul 28 22:44:42.565735 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:44:42.565704    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-dd9c88-5c87c5484d-fgx7f"
Jul 28 22:45:03.186862 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:45:03.186824    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mcj8p_7bab32b0-13c6-4055-916e-a7454bafb5b6/ovn-acl-logging/0.log"
Jul 28 22:45:03.190209 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:45:03.190190    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mcj8p_7bab32b0-13c6-4055-916e-a7454bafb5b6/ovn-acl-logging/0.log"
Jul 28 22:46:31.131941 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:31.131904    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9216b7-547d99479c-bn7dm"]
Jul 28 22:46:31.132644 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:31.132183    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9216b7-547d99479c-bn7dm" podUID="377c437e-7fd1-43ba-8a26-6450f73d708c" containerName="maas-api" containerID="cri-o://53d7c94f76b860db207d19c8c5c1a28c6c072961931a4ac7579d77cb2b02cae2" gracePeriod=30
Jul 28 22:46:31.172820 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:31.172789    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9216b7-6d8f8c4cd8-vzbhn"]
Jul 28 22:46:31.173045 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:31.173019    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-9216b7-6d8f8c4cd8-vzbhn" podUID="b05f2b3d-98f9-4b02-9df7-3952783fb6e8" containerName="payload-processing" containerID="cri-o://5bc1016113c8dd133738e1b7984a080e3f3b9fb9c6a135f4bb4d9c71cd46fa8b" gracePeriod=30
Jul 28 22:46:31.181594 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:31.181569    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9216b7-76866769cb-7c2sl"]
Jul 28 22:46:31.181879 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:31.181851    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9216b7-76866769cb-7c2sl" podUID="6efb5282-8f18-4e60-aa94-287529eeeea4" containerName="payload-pre-processing" containerID="cri-o://35edc575bf4d868161edcb455b713346362e606fcfdbc4347e83cb5358d2257a" gracePeriod=30
Jul 28 22:46:31.450062 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:31.450029    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9216b7-6d8f8c4cd8-vzbhn"
Jul 28 22:46:31.497733 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:31.497708    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9216b7-547d99479c-bn7dm"
Jul 28 22:46:31.500768 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:31.500746    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9216b7-76866769cb-7c2sl"
Jul 28 22:46:31.566196 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:31.566162    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mq9f\" (UniqueName: \"kubernetes.io/projected/b05f2b3d-98f9-4b02-9df7-3952783fb6e8-kube-api-access-8mq9f\") pod \"b05f2b3d-98f9-4b02-9df7-3952783fb6e8\" (UID: \"b05f2b3d-98f9-4b02-9df7-3952783fb6e8\") "
Jul 28 22:46:31.566387 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:31.566211    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b05f2b3d-98f9-4b02-9df7-3952783fb6e8-plugins-config-volume\") pod \"b05f2b3d-98f9-4b02-9df7-3952783fb6e8\" (UID: \"b05f2b3d-98f9-4b02-9df7-3952783fb6e8\") "
Jul 28 22:46:31.566621 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:31.566598    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b05f2b3d-98f9-4b02-9df7-3952783fb6e8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b05f2b3d-98f9-4b02-9df7-3952783fb6e8" (UID: "b05f2b3d-98f9-4b02-9df7-3952783fb6e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:46:31.568257 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:31.568229    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b05f2b3d-98f9-4b02-9df7-3952783fb6e8-kube-api-access-8mq9f" (OuterVolumeSpecName: "kube-api-access-8mq9f") pod "b05f2b3d-98f9-4b02-9df7-3952783fb6e8" (UID: "b05f2b3d-98f9-4b02-9df7-3952783fb6e8"). InnerVolumeSpecName "kube-api-access-8mq9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:46:31.666991 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:31.666902    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6efb5282-8f18-4e60-aa94-287529eeeea4-plugins-config-volume\") pod \"6efb5282-8f18-4e60-aa94-287529eeeea4\" (UID: \"6efb5282-8f18-4e60-aa94-287529eeeea4\") "
Jul 28 22:46:31.667147 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:31.667005    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5lrt\" (UniqueName: \"kubernetes.io/projected/6efb5282-8f18-4e60-aa94-287529eeeea4-kube-api-access-g5lrt\") pod \"6efb5282-8f18-4e60-aa94-287529eeeea4\" (UID: \"6efb5282-8f18-4e60-aa94-287529eeeea4\") "
Jul 28 22:46:31.667147 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:31.667031    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/377c437e-7fd1-43ba-8a26-6450f73d708c-maas-api-tls\") pod \"377c437e-7fd1-43ba-8a26-6450f73d708c\" (UID: \"377c437e-7fd1-43ba-8a26-6450f73d708c\") "
Jul 28 22:46:31.667147 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:31.667087    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2hfx\" (UniqueName: \"kubernetes.io/projected/377c437e-7fd1-43ba-8a26-6450f73d708c-kube-api-access-v2hfx\") pod \"377c437e-7fd1-43ba-8a26-6450f73d708c\" (UID: \"377c437e-7fd1-43ba-8a26-6450f73d708c\") "
Jul 28 22:46:31.667147 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:31.667111    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/377c437e-7fd1-43ba-8a26-6450f73d708c-openshift-service-ca\") pod \"377c437e-7fd1-43ba-8a26-6450f73d708c\" (UID: \"377c437e-7fd1-43ba-8a26-6450f73d708c\") "
Jul 28 22:46:31.667437 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:31.667395    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mq9f\" (UniqueName: \"kubernetes.io/projected/b05f2b3d-98f9-4b02-9df7-3952783fb6e8-kube-api-access-8mq9f\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:46:31.667437 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:31.667408    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6efb5282-8f18-4e60-aa94-287529eeeea4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6efb5282-8f18-4e60-aa94-287529eeeea4" (UID: "6efb5282-8f18-4e60-aa94-287529eeeea4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:46:31.667605 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:31.667420    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b05f2b3d-98f9-4b02-9df7-3952783fb6e8-plugins-config-volume\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:46:31.667676 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:31.667646    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/377c437e-7fd1-43ba-8a26-6450f73d708c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "377c437e-7fd1-43ba-8a26-6450f73d708c" (UID: "377c437e-7fd1-43ba-8a26-6450f73d708c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:46:31.669220 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:31.669190    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/377c437e-7fd1-43ba-8a26-6450f73d708c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "377c437e-7fd1-43ba-8a26-6450f73d708c" (UID: "377c437e-7fd1-43ba-8a26-6450f73d708c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:46:31.669327 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:31.669259    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/377c437e-7fd1-43ba-8a26-6450f73d708c-kube-api-access-v2hfx" (OuterVolumeSpecName: "kube-api-access-v2hfx") pod "377c437e-7fd1-43ba-8a26-6450f73d708c" (UID: "377c437e-7fd1-43ba-8a26-6450f73d708c"). InnerVolumeSpecName "kube-api-access-v2hfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:46:31.669454 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:31.669438    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6efb5282-8f18-4e60-aa94-287529eeeea4-kube-api-access-g5lrt" (OuterVolumeSpecName: "kube-api-access-g5lrt") pod "6efb5282-8f18-4e60-aa94-287529eeeea4" (UID: "6efb5282-8f18-4e60-aa94-287529eeeea4"). InnerVolumeSpecName "kube-api-access-g5lrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:46:31.768401 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:31.768367    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5lrt\" (UniqueName: \"kubernetes.io/projected/6efb5282-8f18-4e60-aa94-287529eeeea4-kube-api-access-g5lrt\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:46:31.768401 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:31.768394    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/377c437e-7fd1-43ba-8a26-6450f73d708c-maas-api-tls\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:46:31.768401 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:31.768405    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2hfx\" (UniqueName: \"kubernetes.io/projected/377c437e-7fd1-43ba-8a26-6450f73d708c-kube-api-access-v2hfx\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:46:31.768617 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:31.768414    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/377c437e-7fd1-43ba-8a26-6450f73d708c-openshift-service-ca\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:46:31.768617 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:31.768422    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6efb5282-8f18-4e60-aa94-287529eeeea4-plugins-config-volume\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:46:32.154684 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:32.154656    2563 generic.go:356] "Generic (PLEG): container finished" podID="377c437e-7fd1-43ba-8a26-6450f73d708c" containerID="53d7c94f76b860db207d19c8c5c1a28c6c072961931a4ac7579d77cb2b02cae2" exitCode=0
Jul 28 22:46:32.155034 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:32.154700    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9216b7-547d99479c-bn7dm"
Jul 28 22:46:32.155034 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:32.154736    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9216b7-547d99479c-bn7dm" event={"ID":"377c437e-7fd1-43ba-8a26-6450f73d708c","Type":"ContainerDied","Data":"53d7c94f76b860db207d19c8c5c1a28c6c072961931a4ac7579d77cb2b02cae2"}
Jul 28 22:46:32.155034 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:32.154782    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9216b7-547d99479c-bn7dm" event={"ID":"377c437e-7fd1-43ba-8a26-6450f73d708c","Type":"ContainerDied","Data":"fa8b7752c5c0e5345e1c515874c676d38965443efd9354fc18fa98b26db81512"}
Jul 28 22:46:32.155034 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:32.154804    2563 scope.go:117] "RemoveContainer" containerID="53d7c94f76b860db207d19c8c5c1a28c6c072961931a4ac7579d77cb2b02cae2"
Jul 28 22:46:32.156097 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:32.156060    2563 generic.go:356] "Generic (PLEG): container finished" podID="b05f2b3d-98f9-4b02-9df7-3952783fb6e8" containerID="5bc1016113c8dd133738e1b7984a080e3f3b9fb9c6a135f4bb4d9c71cd46fa8b" exitCode=0
Jul 28 22:46:32.156184 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:32.156135    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9216b7-6d8f8c4cd8-vzbhn"
Jul 28 22:46:32.156184 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:32.156148    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9216b7-6d8f8c4cd8-vzbhn" event={"ID":"b05f2b3d-98f9-4b02-9df7-3952783fb6e8","Type":"ContainerDied","Data":"5bc1016113c8dd133738e1b7984a080e3f3b9fb9c6a135f4bb4d9c71cd46fa8b"}
Jul 28 22:46:32.156297 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:32.156219    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9216b7-6d8f8c4cd8-vzbhn" event={"ID":"b05f2b3d-98f9-4b02-9df7-3952783fb6e8","Type":"ContainerDied","Data":"773338b9f45e9ffb85b0aff56107a95e3c83bd1aaa5e9fb2556f57a01a7a736b"}
Jul 28 22:46:32.157448 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:32.157423    2563 generic.go:356] "Generic (PLEG): container finished" podID="6efb5282-8f18-4e60-aa94-287529eeeea4" containerID="35edc575bf4d868161edcb455b713346362e606fcfdbc4347e83cb5358d2257a" exitCode=0
Jul 28 22:46:32.157562 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:32.157463    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9216b7-76866769cb-7c2sl" event={"ID":"6efb5282-8f18-4e60-aa94-287529eeeea4","Type":"ContainerDied","Data":"35edc575bf4d868161edcb455b713346362e606fcfdbc4347e83cb5358d2257a"}
Jul 28 22:46:32.157562 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:32.157511    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9216b7-76866769cb-7c2sl" event={"ID":"6efb5282-8f18-4e60-aa94-287529eeeea4","Type":"ContainerDied","Data":"f4990fbf82af459fad8a4534ca44e6e171986fcefdd19131da61418550c161dd"}
Jul 28 22:46:32.157562 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:32.157472    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9216b7-76866769cb-7c2sl"
Jul 28 22:46:32.164878 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:32.164858    2563 scope.go:117] "RemoveContainer" containerID="53d7c94f76b860db207d19c8c5c1a28c6c072961931a4ac7579d77cb2b02cae2"
Jul 28 22:46:32.165190 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:46:32.165168    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53d7c94f76b860db207d19c8c5c1a28c6c072961931a4ac7579d77cb2b02cae2\": container with ID starting with 53d7c94f76b860db207d19c8c5c1a28c6c072961931a4ac7579d77cb2b02cae2 not found: ID does not exist" containerID="53d7c94f76b860db207d19c8c5c1a28c6c072961931a4ac7579d77cb2b02cae2"
Jul 28 22:46:32.165290 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:32.165197    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53d7c94f76b860db207d19c8c5c1a28c6c072961931a4ac7579d77cb2b02cae2"} err="failed to get container status \"53d7c94f76b860db207d19c8c5c1a28c6c072961931a4ac7579d77cb2b02cae2\": rpc error: code = NotFound desc = could not find container \"53d7c94f76b860db207d19c8c5c1a28c6c072961931a4ac7579d77cb2b02cae2\": container with ID starting with 53d7c94f76b860db207d19c8c5c1a28c6c072961931a4ac7579d77cb2b02cae2 not found: ID does not exist"
Jul 28 22:46:32.165290 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:32.165218    2563 scope.go:117] "RemoveContainer" containerID="5bc1016113c8dd133738e1b7984a080e3f3b9fb9c6a135f4bb4d9c71cd46fa8b"
Jul 28 22:46:32.173817 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:32.173799    2563 scope.go:117] "RemoveContainer" containerID="5bc1016113c8dd133738e1b7984a080e3f3b9fb9c6a135f4bb4d9c71cd46fa8b"
Jul 28 22:46:32.174065 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:46:32.174046    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bc1016113c8dd133738e1b7984a080e3f3b9fb9c6a135f4bb4d9c71cd46fa8b\": container with ID starting with 5bc1016113c8dd133738e1b7984a080e3f3b9fb9c6a135f4bb4d9c71cd46fa8b not found: ID does not exist" containerID="5bc1016113c8dd133738e1b7984a080e3f3b9fb9c6a135f4bb4d9c71cd46fa8b"
Jul 28 22:46:32.174130 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:32.174070    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bc1016113c8dd133738e1b7984a080e3f3b9fb9c6a135f4bb4d9c71cd46fa8b"} err="failed to get container status \"5bc1016113c8dd133738e1b7984a080e3f3b9fb9c6a135f4bb4d9c71cd46fa8b\": rpc error: code = NotFound desc = could not find container \"5bc1016113c8dd133738e1b7984a080e3f3b9fb9c6a135f4bb4d9c71cd46fa8b\": container with ID starting with 5bc1016113c8dd133738e1b7984a080e3f3b9fb9c6a135f4bb4d9c71cd46fa8b not found: ID does not exist"
Jul 28 22:46:32.174130 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:32.174085    2563 scope.go:117] "RemoveContainer" containerID="35edc575bf4d868161edcb455b713346362e606fcfdbc4347e83cb5358d2257a"
Jul 28 22:46:32.182186 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:32.182160    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9216b7-547d99479c-bn7dm"]
Jul 28 22:46:32.182942 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:32.182888    2563 scope.go:117] "RemoveContainer" containerID="35edc575bf4d868161edcb455b713346362e606fcfdbc4347e83cb5358d2257a"
Jul 28 22:46:32.183227 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:46:32.183158    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35edc575bf4d868161edcb455b713346362e606fcfdbc4347e83cb5358d2257a\": container with ID starting with 35edc575bf4d868161edcb455b713346362e606fcfdbc4347e83cb5358d2257a not found: ID does not exist" containerID="35edc575bf4d868161edcb455b713346362e606fcfdbc4347e83cb5358d2257a"
Jul 28 22:46:32.183227 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:32.183187    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35edc575bf4d868161edcb455b713346362e606fcfdbc4347e83cb5358d2257a"} err="failed to get container status \"35edc575bf4d868161edcb455b713346362e606fcfdbc4347e83cb5358d2257a\": rpc error: code = NotFound desc = could not find container \"35edc575bf4d868161edcb455b713346362e606fcfdbc4347e83cb5358d2257a\": container with ID starting with 35edc575bf4d868161edcb455b713346362e606fcfdbc4347e83cb5358d2257a not found: ID does not exist"
Jul 28 22:46:32.184912 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:32.184894    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9216b7-547d99479c-bn7dm"]
Jul 28 22:46:32.194950 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:32.194922    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9216b7-6d8f8c4cd8-vzbhn"]
Jul 28 22:46:32.198408 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:32.198390    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9216b7-6d8f8c4cd8-vzbhn"]
Jul 28 22:46:32.211451 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:32.211429    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9216b7-76866769cb-7c2sl"]
Jul 28 22:46:32.215090 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:32.215072    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9216b7-76866769cb-7c2sl"]
Jul 28 22:46:33.130137 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:33.130104    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="377c437e-7fd1-43ba-8a26-6450f73d708c" path="/var/lib/kubelet/pods/377c437e-7fd1-43ba-8a26-6450f73d708c/volumes"
Jul 28 22:46:33.130498 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:33.130483    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6efb5282-8f18-4e60-aa94-287529eeeea4" path="/var/lib/kubelet/pods/6efb5282-8f18-4e60-aa94-287529eeeea4/volumes"
Jul 28 22:46:33.130801 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:33.130790    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b05f2b3d-98f9-4b02-9df7-3952783fb6e8" path="/var/lib/kubelet/pods/b05f2b3d-98f9-4b02-9df7-3952783fb6e8/volumes"
Jul 28 22:46:48.498635 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:48.498599    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dd9c88-558f6b84d5-rc5hv"]
Jul 28 22:46:48.501429 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:48.498837    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd9c88-558f6b84d5-rc5hv" podUID="0a9226b1-f3c5-4fc9-a127-c5e2c6708725" containerName="maas-api" containerID="cri-o://e8f83e71f268af35a10078dff5a19df1ac43bfe64ef09a2c1130cab5cd24aac2" gracePeriod=30
Jul 28 22:46:48.561594 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:48.561562    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dd9c88-5c87c5484d-fgx7f"]
Jul 28 22:46:48.561845 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:48.561793    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-dd9c88-5c87c5484d-fgx7f" podUID="658ba086-c4f5-4911-af7a-1ac6bc17a47c" containerName="payload-processing" containerID="cri-o://91d4c13b73f184dbe05ef4491047ab2332301a915079614faa3c7cba153e30de" gracePeriod=30
Jul 28 22:46:48.857756 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:48.857728    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-dd9c88-5c87c5484d-fgx7f"
Jul 28 22:46:48.926276 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:48.926248    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6rt5\" (UniqueName: \"kubernetes.io/projected/658ba086-c4f5-4911-af7a-1ac6bc17a47c-kube-api-access-b6rt5\") pod \"658ba086-c4f5-4911-af7a-1ac6bc17a47c\" (UID: \"658ba086-c4f5-4911-af7a-1ac6bc17a47c\") "
Jul 28 22:46:48.926475 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:48.926346    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/658ba086-c4f5-4911-af7a-1ac6bc17a47c-plugins-config-volume\") pod \"658ba086-c4f5-4911-af7a-1ac6bc17a47c\" (UID: \"658ba086-c4f5-4911-af7a-1ac6bc17a47c\") "
Jul 28 22:46:48.926737 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:48.926709    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/658ba086-c4f5-4911-af7a-1ac6bc17a47c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "658ba086-c4f5-4911-af7a-1ac6bc17a47c" (UID: "658ba086-c4f5-4911-af7a-1ac6bc17a47c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:46:48.929419 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:48.929395    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/658ba086-c4f5-4911-af7a-1ac6bc17a47c-kube-api-access-b6rt5" (OuterVolumeSpecName: "kube-api-access-b6rt5") pod "658ba086-c4f5-4911-af7a-1ac6bc17a47c" (UID: "658ba086-c4f5-4911-af7a-1ac6bc17a47c"). InnerVolumeSpecName "kube-api-access-b6rt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:46:48.956447 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:48.956424    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd9c88-558f6b84d5-rc5hv"
Jul 28 22:46:49.026893 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:49.026863    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9226b1-f3c5-4fc9-a127-c5e2c6708725-openshift-service-ca\") pod \"0a9226b1-f3c5-4fc9-a127-c5e2c6708725\" (UID: \"0a9226b1-f3c5-4fc9-a127-c5e2c6708725\") "
Jul 28 22:46:49.027044 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:49.026950    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a9226b1-f3c5-4fc9-a127-c5e2c6708725-maas-api-tls\") pod \"0a9226b1-f3c5-4fc9-a127-c5e2c6708725\" (UID: \"0a9226b1-f3c5-4fc9-a127-c5e2c6708725\") "
Jul 28 22:46:49.027044 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:49.026975    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6n8jx\" (UniqueName: \"kubernetes.io/projected/0a9226b1-f3c5-4fc9-a127-c5e2c6708725-kube-api-access-6n8jx\") pod \"0a9226b1-f3c5-4fc9-a127-c5e2c6708725\" (UID: \"0a9226b1-f3c5-4fc9-a127-c5e2c6708725\") "
Jul 28 22:46:49.027137 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:49.027124    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6rt5\" (UniqueName: \"kubernetes.io/projected/658ba086-c4f5-4911-af7a-1ac6bc17a47c-kube-api-access-b6rt5\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:46:49.027202 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:49.027139    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/658ba086-c4f5-4911-af7a-1ac6bc17a47c-plugins-config-volume\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:46:49.027202 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:49.027178    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a9226b1-f3c5-4fc9-a127-c5e2c6708725-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0a9226b1-f3c5-4fc9-a127-c5e2c6708725" (UID: "0a9226b1-f3c5-4fc9-a127-c5e2c6708725"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:46:49.029035 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:49.029003    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a9226b1-f3c5-4fc9-a127-c5e2c6708725-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0a9226b1-f3c5-4fc9-a127-c5e2c6708725" (UID: "0a9226b1-f3c5-4fc9-a127-c5e2c6708725"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:46:49.029035 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:49.029008    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9226b1-f3c5-4fc9-a127-c5e2c6708725-kube-api-access-6n8jx" (OuterVolumeSpecName: "kube-api-access-6n8jx") pod "0a9226b1-f3c5-4fc9-a127-c5e2c6708725" (UID: "0a9226b1-f3c5-4fc9-a127-c5e2c6708725"). InnerVolumeSpecName "kube-api-access-6n8jx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:46:49.128108 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:49.128034    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a9226b1-f3c5-4fc9-a127-c5e2c6708725-maas-api-tls\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:46:49.128108 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:49.128067    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n8jx\" (UniqueName: \"kubernetes.io/projected/0a9226b1-f3c5-4fc9-a127-c5e2c6708725-kube-api-access-6n8jx\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:46:49.128108 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:49.128083    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9226b1-f3c5-4fc9-a127-c5e2c6708725-openshift-service-ca\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:46:49.239950 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:49.239916    2563 generic.go:356] "Generic (PLEG): container finished" podID="658ba086-c4f5-4911-af7a-1ac6bc17a47c" containerID="91d4c13b73f184dbe05ef4491047ab2332301a915079614faa3c7cba153e30de" exitCode=0
Jul 28 22:46:49.240114 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:49.239962    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-dd9c88-5c87c5484d-fgx7f"
Jul 28 22:46:49.240114 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:49.240003    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-dd9c88-5c87c5484d-fgx7f" event={"ID":"658ba086-c4f5-4911-af7a-1ac6bc17a47c","Type":"ContainerDied","Data":"91d4c13b73f184dbe05ef4491047ab2332301a915079614faa3c7cba153e30de"}
Jul 28 22:46:49.240114 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:49.240051    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-dd9c88-5c87c5484d-fgx7f" event={"ID":"658ba086-c4f5-4911-af7a-1ac6bc17a47c","Type":"ContainerDied","Data":"4fb34edce2a527c8f5ec468d9a46a5b96af6833bced3eb5506175f3b91217360"}
Jul 28 22:46:49.240114 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:49.240077    2563 scope.go:117] "RemoveContainer" containerID="91d4c13b73f184dbe05ef4491047ab2332301a915079614faa3c7cba153e30de"
Jul 28 22:46:49.241414 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:49.241385    2563 generic.go:356] "Generic (PLEG): container finished" podID="0a9226b1-f3c5-4fc9-a127-c5e2c6708725" containerID="e8f83e71f268af35a10078dff5a19df1ac43bfe64ef09a2c1130cab5cd24aac2" exitCode=0
Jul 28 22:46:49.241539 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:49.241415    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd9c88-558f6b84d5-rc5hv" event={"ID":"0a9226b1-f3c5-4fc9-a127-c5e2c6708725","Type":"ContainerDied","Data":"e8f83e71f268af35a10078dff5a19df1ac43bfe64ef09a2c1130cab5cd24aac2"}
Jul 28 22:46:49.241539 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:49.241439    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd9c88-558f6b84d5-rc5hv"
Jul 28 22:46:49.241539 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:49.241456    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd9c88-558f6b84d5-rc5hv" event={"ID":"0a9226b1-f3c5-4fc9-a127-c5e2c6708725","Type":"ContainerDied","Data":"f140699a0e9a5662559f295a1f6d8fec511da3dcd8c17d588b604f20b2959396"}
Jul 28 22:46:49.249058 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:49.249041    2563 scope.go:117] "RemoveContainer" containerID="91d4c13b73f184dbe05ef4491047ab2332301a915079614faa3c7cba153e30de"
Jul 28 22:46:49.249324 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:46:49.249294    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91d4c13b73f184dbe05ef4491047ab2332301a915079614faa3c7cba153e30de\": container with ID starting with 91d4c13b73f184dbe05ef4491047ab2332301a915079614faa3c7cba153e30de not found: ID does not exist" containerID="91d4c13b73f184dbe05ef4491047ab2332301a915079614faa3c7cba153e30de"
Jul 28 22:46:49.249450 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:49.249329    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91d4c13b73f184dbe05ef4491047ab2332301a915079614faa3c7cba153e30de"} err="failed to get container status \"91d4c13b73f184dbe05ef4491047ab2332301a915079614faa3c7cba153e30de\": rpc error: code = NotFound desc = could not find container \"91d4c13b73f184dbe05ef4491047ab2332301a915079614faa3c7cba153e30de\": container with ID starting with 91d4c13b73f184dbe05ef4491047ab2332301a915079614faa3c7cba153e30de not found: ID does not exist"
Jul 28 22:46:49.249450 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:49.249345    2563 scope.go:117] "RemoveContainer" containerID="e8f83e71f268af35a10078dff5a19df1ac43bfe64ef09a2c1130cab5cd24aac2"
Jul 28 22:46:49.256858 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:49.256826    2563 scope.go:117] "RemoveContainer" containerID="e8f83e71f268af35a10078dff5a19df1ac43bfe64ef09a2c1130cab5cd24aac2"
Jul 28 22:46:49.257119 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:46:49.257102    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8f83e71f268af35a10078dff5a19df1ac43bfe64ef09a2c1130cab5cd24aac2\": container with ID starting with e8f83e71f268af35a10078dff5a19df1ac43bfe64ef09a2c1130cab5cd24aac2 not found: ID does not exist" containerID="e8f83e71f268af35a10078dff5a19df1ac43bfe64ef09a2c1130cab5cd24aac2"
Jul 28 22:46:49.257178 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:49.257124    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8f83e71f268af35a10078dff5a19df1ac43bfe64ef09a2c1130cab5cd24aac2"} err="failed to get container status \"e8f83e71f268af35a10078dff5a19df1ac43bfe64ef09a2c1130cab5cd24aac2\": rpc error: code = NotFound desc = could not find container \"e8f83e71f268af35a10078dff5a19df1ac43bfe64ef09a2c1130cab5cd24aac2\": container with ID starting with e8f83e71f268af35a10078dff5a19df1ac43bfe64ef09a2c1130cab5cd24aac2 not found: ID does not exist"
Jul 28 22:46:49.263894 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:49.263868    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dd9c88-558f6b84d5-rc5hv"]
Jul 28 22:46:49.266534 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:49.266513    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dd9c88-558f6b84d5-rc5hv"]
Jul 28 22:46:49.283195 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:49.283172    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dd9c88-5c87c5484d-fgx7f"]
Jul 28 22:46:49.286666 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:49.286645    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dd9c88-5c87c5484d-fgx7f"]
Jul 28 22:46:51.129705 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:51.129670    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a9226b1-f3c5-4fc9-a127-c5e2c6708725" path="/var/lib/kubelet/pods/0a9226b1-f3c5-4fc9-a127-c5e2c6708725/volumes"
Jul 28 22:46:51.130092 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:51.130023    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="658ba086-c4f5-4911-af7a-1ac6bc17a47c" path="/var/lib/kubelet/pods/658ba086-c4f5-4911-af7a-1ac6bc17a47c/volumes"
Jul 28 22:46:59.498003 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:59.497923    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-26e94d-665b4fdb49-tvprt"]
Jul 28 22:46:59.498404 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:59.498273    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a9226b1-f3c5-4fc9-a127-c5e2c6708725" containerName="maas-api"
Jul 28 22:46:59.498404 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:59.498286    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a9226b1-f3c5-4fc9-a127-c5e2c6708725" containerName="maas-api"
Jul 28 22:46:59.498404 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:59.498302    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="658ba086-c4f5-4911-af7a-1ac6bc17a47c" containerName="payload-processing"
Jul 28 22:46:59.498404 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:59.498308    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="658ba086-c4f5-4911-af7a-1ac6bc17a47c" containerName="payload-processing"
Jul 28 22:46:59.498404 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:59.498322    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b05f2b3d-98f9-4b02-9df7-3952783fb6e8" containerName="payload-processing"
Jul 28 22:46:59.498404 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:59.498327    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b05f2b3d-98f9-4b02-9df7-3952783fb6e8" containerName="payload-processing"
Jul 28 22:46:59.498404 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:59.498335    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6efb5282-8f18-4e60-aa94-287529eeeea4" containerName="payload-pre-processing"
Jul 28 22:46:59.498404 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:59.498340    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6efb5282-8f18-4e60-aa94-287529eeeea4" containerName="payload-pre-processing"
Jul 28 22:46:59.498404 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:59.498372    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="377c437e-7fd1-43ba-8a26-6450f73d708c" containerName="maas-api"
Jul 28 22:46:59.498404 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:59.498378    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="377c437e-7fd1-43ba-8a26-6450f73d708c" containerName="maas-api"
Jul 28 22:46:59.498726 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:59.498437    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="377c437e-7fd1-43ba-8a26-6450f73d708c" containerName="maas-api"
Jul 28 22:46:59.498726 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:59.498445    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="658ba086-c4f5-4911-af7a-1ac6bc17a47c" containerName="payload-processing"
Jul 28 22:46:59.498726 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:59.498452    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6efb5282-8f18-4e60-aa94-287529eeeea4" containerName="payload-pre-processing"
Jul 28 22:46:59.498726 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:59.498457    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a9226b1-f3c5-4fc9-a127-c5e2c6708725" containerName="maas-api"
Jul 28 22:46:59.498726 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:59.498463    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b05f2b3d-98f9-4b02-9df7-3952783fb6e8" containerName="payload-processing"
Jul 28 22:46:59.502970 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:59.502950    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26e94d-665b4fdb49-tvprt"
Jul 28 22:46:59.505737 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:59.505710    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-26e94d-serving-cert\""
Jul 28 22:46:59.512807 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:59.512783    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-26e94d-665b4fdb49-tvprt"]
Jul 28 22:46:59.620881 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:59.620841    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bbf6abb-c937-4ce9-8ba6-c02af67e6983-openshift-service-ca\") pod \"maas-api-e2e-api-a-26e94d-665b4fdb49-tvprt\" (UID: \"6bbf6abb-c937-4ce9-8ba6-c02af67e6983\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26e94d-665b4fdb49-tvprt"
Jul 28 22:46:59.621073 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:59.620897    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w8vw\" (UniqueName: \"kubernetes.io/projected/6bbf6abb-c937-4ce9-8ba6-c02af67e6983-kube-api-access-9w8vw\") pod \"maas-api-e2e-api-a-26e94d-665b4fdb49-tvprt\" (UID: \"6bbf6abb-c937-4ce9-8ba6-c02af67e6983\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26e94d-665b4fdb49-tvprt"
Jul 28 22:46:59.621073 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:59.620928    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bbf6abb-c937-4ce9-8ba6-c02af67e6983-maas-api-tls\") pod \"maas-api-e2e-api-a-26e94d-665b4fdb49-tvprt\" (UID: \"6bbf6abb-c937-4ce9-8ba6-c02af67e6983\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26e94d-665b4fdb49-tvprt"
Jul 28 22:46:59.721820 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:59.721783    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bbf6abb-c937-4ce9-8ba6-c02af67e6983-openshift-service-ca\") pod \"maas-api-e2e-api-a-26e94d-665b4fdb49-tvprt\" (UID: \"6bbf6abb-c937-4ce9-8ba6-c02af67e6983\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26e94d-665b4fdb49-tvprt"
Jul 28 22:46:59.721820 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:59.721822    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w8vw\" (UniqueName: \"kubernetes.io/projected/6bbf6abb-c937-4ce9-8ba6-c02af67e6983-kube-api-access-9w8vw\") pod \"maas-api-e2e-api-a-26e94d-665b4fdb49-tvprt\" (UID: \"6bbf6abb-c937-4ce9-8ba6-c02af67e6983\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26e94d-665b4fdb49-tvprt"
Jul 28 22:46:59.722026 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:59.721843    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bbf6abb-c937-4ce9-8ba6-c02af67e6983-maas-api-tls\") pod \"maas-api-e2e-api-a-26e94d-665b4fdb49-tvprt\" (UID: \"6bbf6abb-c937-4ce9-8ba6-c02af67e6983\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26e94d-665b4fdb49-tvprt"
Jul 28 22:46:59.722026 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:46:59.721969    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-26e94d-serving-cert: secret "maas-api-e2e-api-a-26e94d-serving-cert" not found
Jul 28 22:46:59.722026 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:46:59.722024    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bbf6abb-c937-4ce9-8ba6-c02af67e6983-maas-api-tls podName:6bbf6abb-c937-4ce9-8ba6-c02af67e6983 nodeName:}" failed. No retries permitted until 2026-07-28 22:47:00.222008327 +0000 UTC m=+2517.602145950 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6bbf6abb-c937-4ce9-8ba6-c02af67e6983-maas-api-tls") pod "maas-api-e2e-api-a-26e94d-665b4fdb49-tvprt" (UID: "6bbf6abb-c937-4ce9-8ba6-c02af67e6983") : secret "maas-api-e2e-api-a-26e94d-serving-cert" not found
Jul 28 22:46:59.722520 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:59.722500    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bbf6abb-c937-4ce9-8ba6-c02af67e6983-openshift-service-ca\") pod \"maas-api-e2e-api-a-26e94d-665b4fdb49-tvprt\" (UID: \"6bbf6abb-c937-4ce9-8ba6-c02af67e6983\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26e94d-665b4fdb49-tvprt"
Jul 28 22:46:59.735709 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:46:59.735675    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w8vw\" (UniqueName: \"kubernetes.io/projected/6bbf6abb-c937-4ce9-8ba6-c02af67e6983-kube-api-access-9w8vw\") pod \"maas-api-e2e-api-a-26e94d-665b4fdb49-tvprt\" (UID: \"6bbf6abb-c937-4ce9-8ba6-c02af67e6983\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26e94d-665b4fdb49-tvprt"
Jul 28 22:47:00.227192 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:00.227157    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bbf6abb-c937-4ce9-8ba6-c02af67e6983-maas-api-tls\") pod \"maas-api-e2e-api-a-26e94d-665b4fdb49-tvprt\" (UID: \"6bbf6abb-c937-4ce9-8ba6-c02af67e6983\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26e94d-665b4fdb49-tvprt"
Jul 28 22:47:00.229517 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:00.229488    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bbf6abb-c937-4ce9-8ba6-c02af67e6983-maas-api-tls\") pod \"maas-api-e2e-api-a-26e94d-665b4fdb49-tvprt\" (UID: \"6bbf6abb-c937-4ce9-8ba6-c02af67e6983\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26e94d-665b4fdb49-tvprt"
Jul 28 22:47:00.280507 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:00.280469    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-26e94d-cbdd7fff6-hb7r9"]
Jul 28 22:47:00.284037 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:00.284016    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-26e94d-cbdd7fff6-hb7r9"
Jul 28 22:47:00.286605 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:00.286585    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-26e94d\""
Jul 28 22:47:00.286721 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:00.286586    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-26e94d-dockercfg-pqxrm\""
Jul 28 22:47:00.300537 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:00.300514    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-26e94d-cbdd7fff6-hb7r9"]
Jul 28 22:47:00.320129 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:00.320097    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-26e94d-5bb8f9dcb8-78zpb"]
Jul 28 22:47:00.323450 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:00.323434    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-26e94d-5bb8f9dcb8-78zpb"
Jul 28 22:47:00.334381 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:00.334343    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-26e94d-5bb8f9dcb8-78zpb"]
Jul 28 22:47:00.413276 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:00.413246    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26e94d-665b4fdb49-tvprt"
Jul 28 22:47:00.428830 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:00.428795    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2ld7\" (UniqueName: \"kubernetes.io/projected/d5a34a78-5df1-465c-9a98-adf40ab144d0-kube-api-access-b2ld7\") pod \"payload-pre-processing-e2e-api-a-26e94d-cbdd7fff6-hb7r9\" (UID: \"d5a34a78-5df1-465c-9a98-adf40ab144d0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-26e94d-cbdd7fff6-hb7r9"
Jul 28 22:47:00.429059 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:00.429034    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5a34a78-5df1-465c-9a98-adf40ab144d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-26e94d-cbdd7fff6-hb7r9\" (UID: \"d5a34a78-5df1-465c-9a98-adf40ab144d0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-26e94d-cbdd7fff6-hb7r9"
Jul 28 22:47:00.429203 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:00.429186    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bbx6\" (UniqueName: \"kubernetes.io/projected/39aedb24-3827-4baf-84e3-6011dcf98e2b-kube-api-access-2bbx6\") pod \"payload-processing-e2e-api-a-26e94d-5bb8f9dcb8-78zpb\" (UID: \"39aedb24-3827-4baf-84e3-6011dcf98e2b\") " pod="openshift-ingress/payload-processing-e2e-api-a-26e94d-5bb8f9dcb8-78zpb"
Jul 28 22:47:00.429460 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:00.429436    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39aedb24-3827-4baf-84e3-6011dcf98e2b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-26e94d-5bb8f9dcb8-78zpb\" (UID: \"39aedb24-3827-4baf-84e3-6011dcf98e2b\") " pod="openshift-ingress/payload-processing-e2e-api-a-26e94d-5bb8f9dcb8-78zpb"
Jul 28 22:47:00.530989 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:00.530948    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2ld7\" (UniqueName: \"kubernetes.io/projected/d5a34a78-5df1-465c-9a98-adf40ab144d0-kube-api-access-b2ld7\") pod \"payload-pre-processing-e2e-api-a-26e94d-cbdd7fff6-hb7r9\" (UID: \"d5a34a78-5df1-465c-9a98-adf40ab144d0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-26e94d-cbdd7fff6-hb7r9"
Jul 28 22:47:00.531440 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:00.530999    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5a34a78-5df1-465c-9a98-adf40ab144d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-26e94d-cbdd7fff6-hb7r9\" (UID: \"d5a34a78-5df1-465c-9a98-adf40ab144d0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-26e94d-cbdd7fff6-hb7r9"
Jul 28 22:47:00.531440 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:00.531033    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bbx6\" (UniqueName: \"kubernetes.io/projected/39aedb24-3827-4baf-84e3-6011dcf98e2b-kube-api-access-2bbx6\") pod \"payload-processing-e2e-api-a-26e94d-5bb8f9dcb8-78zpb\" (UID: \"39aedb24-3827-4baf-84e3-6011dcf98e2b\") " pod="openshift-ingress/payload-processing-e2e-api-a-26e94d-5bb8f9dcb8-78zpb"
Jul 28 22:47:00.531440 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:00.531071    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39aedb24-3827-4baf-84e3-6011dcf98e2b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-26e94d-5bb8f9dcb8-78zpb\" (UID: \"39aedb24-3827-4baf-84e3-6011dcf98e2b\") " pod="openshift-ingress/payload-processing-e2e-api-a-26e94d-5bb8f9dcb8-78zpb"
Jul 28 22:47:00.531912 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:00.531893    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39aedb24-3827-4baf-84e3-6011dcf98e2b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-26e94d-5bb8f9dcb8-78zpb\" (UID: \"39aedb24-3827-4baf-84e3-6011dcf98e2b\") " pod="openshift-ingress/payload-processing-e2e-api-a-26e94d-5bb8f9dcb8-78zpb"
Jul 28 22:47:00.531996 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:00.531974    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5a34a78-5df1-465c-9a98-adf40ab144d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-26e94d-cbdd7fff6-hb7r9\" (UID: \"d5a34a78-5df1-465c-9a98-adf40ab144d0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-26e94d-cbdd7fff6-hb7r9"
Jul 28 22:47:00.539654 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:00.539626    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2ld7\" (UniqueName: \"kubernetes.io/projected/d5a34a78-5df1-465c-9a98-adf40ab144d0-kube-api-access-b2ld7\") pod \"payload-pre-processing-e2e-api-a-26e94d-cbdd7fff6-hb7r9\" (UID: \"d5a34a78-5df1-465c-9a98-adf40ab144d0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-26e94d-cbdd7fff6-hb7r9"
Jul 28 22:47:00.539805 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:00.539788    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bbx6\" (UniqueName: \"kubernetes.io/projected/39aedb24-3827-4baf-84e3-6011dcf98e2b-kube-api-access-2bbx6\") pod \"payload-processing-e2e-api-a-26e94d-5bb8f9dcb8-78zpb\" (UID: \"39aedb24-3827-4baf-84e3-6011dcf98e2b\") " pod="openshift-ingress/payload-processing-e2e-api-a-26e94d-5bb8f9dcb8-78zpb"
Jul 28 22:47:00.565507 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:00.565481    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-26e94d-665b4fdb49-tvprt"]
Jul 28 22:47:00.567685 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:47:00.567654    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bbf6abb_c937_4ce9_8ba6_c02af67e6983.slice/crio-8a9b0475847ddea3d9fe6b31652d471f754dd936f4d8abafecc23c4c8c6f3cfb WatchSource:0}: Error finding container 8a9b0475847ddea3d9fe6b31652d471f754dd936f4d8abafecc23c4c8c6f3cfb: Status 404 returned error can't find the container with id 8a9b0475847ddea3d9fe6b31652d471f754dd936f4d8abafecc23c4c8c6f3cfb
Jul 28 22:47:00.569108 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:00.569091    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 22:47:00.595252 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:00.595229    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-26e94d-cbdd7fff6-hb7r9"
Jul 28 22:47:00.634298 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:00.634261    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-26e94d-5bb8f9dcb8-78zpb"
Jul 28 22:47:00.745830 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:00.745644    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-26e94d-cbdd7fff6-hb7r9"]
Jul 28 22:47:00.752061 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:47:00.752026    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a34a78_5df1_465c_9a98_adf40ab144d0.slice/crio-07eaf0d32b60660c93e2c7dee0fb28cb0936a79cc756890e31869943b34af0fc WatchSource:0}: Error finding container 07eaf0d32b60660c93e2c7dee0fb28cb0936a79cc756890e31869943b34af0fc: Status 404 returned error can't find the container with id 07eaf0d32b60660c93e2c7dee0fb28cb0936a79cc756890e31869943b34af0fc
Jul 28 22:47:00.794954 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:00.794925    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-26e94d-5bb8f9dcb8-78zpb"]
Jul 28 22:47:00.796667 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:47:00.796642    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39aedb24_3827_4baf_84e3_6011dcf98e2b.slice/crio-4974b5c6ad684ec52a5744ba545f9a912bdca864cfb4e82354f803de1d39279c WatchSource:0}: Error finding container 4974b5c6ad684ec52a5744ba545f9a912bdca864cfb4e82354f803de1d39279c: Status 404 returned error can't find the container with id 4974b5c6ad684ec52a5744ba545f9a912bdca864cfb4e82354f803de1d39279c
Jul 28 22:47:01.299477 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:01.299378    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-26e94d-5bb8f9dcb8-78zpb" event={"ID":"39aedb24-3827-4baf-84e3-6011dcf98e2b","Type":"ContainerStarted","Data":"f9b281f4acc6ac5d84339e62ead9923064c3dda4347e4f80a6ceb4ef56d6d651"}
Jul 28 22:47:01.299477 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:01.299444    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-26e94d-5bb8f9dcb8-78zpb" event={"ID":"39aedb24-3827-4baf-84e3-6011dcf98e2b","Type":"ContainerStarted","Data":"4974b5c6ad684ec52a5744ba545f9a912bdca864cfb4e82354f803de1d39279c"}
Jul 28 22:47:01.299740 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:01.299607    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-26e94d-5bb8f9dcb8-78zpb"
Jul 28 22:47:01.301059 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:01.301032    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26e94d-665b4fdb49-tvprt" event={"ID":"6bbf6abb-c937-4ce9-8ba6-c02af67e6983","Type":"ContainerStarted","Data":"790d46358f833c3174b4515fdc540a166768883f4531130a03c8a504ce13aacc"}
Jul 28 22:47:01.301186 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:01.301083    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26e94d-665b4fdb49-tvprt"
Jul 28 22:47:01.301186 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:01.301107    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26e94d-665b4fdb49-tvprt" event={"ID":"6bbf6abb-c937-4ce9-8ba6-c02af67e6983","Type":"ContainerStarted","Data":"8a9b0475847ddea3d9fe6b31652d471f754dd936f4d8abafecc23c4c8c6f3cfb"}
Jul 28 22:47:01.302806 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:01.302784    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-26e94d-cbdd7fff6-hb7r9" event={"ID":"d5a34a78-5df1-465c-9a98-adf40ab144d0","Type":"ContainerStarted","Data":"88cbd4669adeea3024979c407acbe22eb94d1ad4c4b1dbc892541ee71061e33c"}
Jul 28 22:47:01.302927 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:01.302833    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-26e94d-cbdd7fff6-hb7r9"
Jul 28 22:47:01.302927 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:01.302855    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-26e94d-cbdd7fff6-hb7r9" event={"ID":"d5a34a78-5df1-465c-9a98-adf40ab144d0","Type":"ContainerStarted","Data":"07eaf0d32b60660c93e2c7dee0fb28cb0936a79cc756890e31869943b34af0fc"}
Jul 28 22:47:01.315613 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:01.315568    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-26e94d-5bb8f9dcb8-78zpb" podStartSLOduration=1.315555026 podStartE2EDuration="1.315555026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:47:01.31425355 +0000 UTC m=+2518.694391206" watchObservedRunningTime="2026-07-28 22:47:01.315555026 +0000 UTC m=+2518.695692669"
Jul 28 22:47:01.329291 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:01.329242    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-26e94d-cbdd7fff6-hb7r9" podStartSLOduration=1.329227937 podStartE2EDuration="1.329227937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:47:01.32783151 +0000 UTC m=+2518.707969155" watchObservedRunningTime="2026-07-28 22:47:01.329227937 +0000 UTC m=+2518.709365581"
Jul 28 22:47:01.345251 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:01.345197    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26e94d-665b4fdb49-tvprt" podStartSLOduration=1.8949954610000002 podStartE2EDuration="2.345179906s" totalImagesPullingTime="450.184445ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:46:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:47:01.34377235 +0000 UTC m=+2518.723909995" watchObservedRunningTime="2026-07-28 22:47:01.345179906 +0000 UTC m=+2518.725317551"
Jul 28 22:47:03.347517 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:03.347481    2563 scope.go:117] "RemoveContainer" containerID="b92e5432b2b4f5a583d24afdb5bc5312a1854483ca1ec89400998ed1cd0ae080"
Jul 28 22:47:07.313323 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:07.313295    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26e94d-665b4fdb49-tvprt"
Jul 28 22:47:12.309886 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:12.309854    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-26e94d-cbdd7fff6-hb7r9"
Jul 28 22:47:12.310281 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:12.309941    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-26e94d-5bb8f9dcb8-78zpb"
Jul 28 22:47:28.991042 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:28.991004    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9bcb6e-5b5dbf7775-bcjqg"]
Jul 28 22:47:28.994779 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:28.994754    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9bcb6e-5b5dbf7775-bcjqg"
Jul 28 22:47:28.997405 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:28.997347    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-9bcb6e-serving-cert\""
Jul 28 22:47:29.005051 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:29.005024    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9bcb6e-5b5dbf7775-bcjqg"]
Jul 28 22:47:29.085667 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:29.085634    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90-openshift-service-ca\") pod \"maas-api-e2e-api-b-9bcb6e-5b5dbf7775-bcjqg\" (UID: \"dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9bcb6e-5b5dbf7775-bcjqg"
Jul 28 22:47:29.085840 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:29.085705    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp8p8\" (UniqueName: \"kubernetes.io/projected/dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90-kube-api-access-lp8p8\") pod \"maas-api-e2e-api-b-9bcb6e-5b5dbf7775-bcjqg\" (UID: \"dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9bcb6e-5b5dbf7775-bcjqg"
Jul 28 22:47:29.085840 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:29.085738    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90-maas-api-tls\") pod \"maas-api-e2e-api-b-9bcb6e-5b5dbf7775-bcjqg\" (UID: \"dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9bcb6e-5b5dbf7775-bcjqg"
Jul 28 22:47:29.186205 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:29.186171    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90-openshift-service-ca\") pod \"maas-api-e2e-api-b-9bcb6e-5b5dbf7775-bcjqg\" (UID: \"dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9bcb6e-5b5dbf7775-bcjqg"
Jul 28 22:47:29.186205 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:29.186212    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp8p8\" (UniqueName: \"kubernetes.io/projected/dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90-kube-api-access-lp8p8\") pod \"maas-api-e2e-api-b-9bcb6e-5b5dbf7775-bcjqg\" (UID: \"dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9bcb6e-5b5dbf7775-bcjqg"
Jul 28 22:47:29.186492 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:29.186239    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90-maas-api-tls\") pod \"maas-api-e2e-api-b-9bcb6e-5b5dbf7775-bcjqg\" (UID: \"dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9bcb6e-5b5dbf7775-bcjqg"
Jul 28 22:47:29.186941 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:29.186913    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90-openshift-service-ca\") pod \"maas-api-e2e-api-b-9bcb6e-5b5dbf7775-bcjqg\" (UID: \"dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9bcb6e-5b5dbf7775-bcjqg"
Jul 28 22:47:29.188667 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:29.188648    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90-maas-api-tls\") pod \"maas-api-e2e-api-b-9bcb6e-5b5dbf7775-bcjqg\" (UID: \"dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9bcb6e-5b5dbf7775-bcjqg"
Jul 28 22:47:29.193617 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:29.193594    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp8p8\" (UniqueName: \"kubernetes.io/projected/dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90-kube-api-access-lp8p8\") pod \"maas-api-e2e-api-b-9bcb6e-5b5dbf7775-bcjqg\" (UID: \"dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9bcb6e-5b5dbf7775-bcjqg"
Jul 28 22:47:29.307386 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:29.307274    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9bcb6e-5b5dbf7775-bcjqg"
Jul 28 22:47:29.453373 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:29.453332    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9bcb6e-5b5dbf7775-bcjqg"]
Jul 28 22:47:29.455393 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:47:29.455333    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc2a2ccc_ae34_4bb5_bfa7_3e0a5fbe5e90.slice/crio-47647d771980cb863906120fa0858642d9c1f166e5c5512ac4fad24863c393df WatchSource:0}: Error finding container 47647d771980cb863906120fa0858642d9c1f166e5c5512ac4fad24863c393df: Status 404 returned error can't find the container with id 47647d771980cb863906120fa0858642d9c1f166e5c5512ac4fad24863c393df
Jul 28 22:47:29.757085 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:29.757052    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9bcb6e-6df6f9fc64-jjh7q"]
Jul 28 22:47:29.761833 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:29.761812    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9bcb6e-6df6f9fc64-jjh7q"
Jul 28 22:47:29.764255 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:29.764234    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-9bcb6e-dockercfg-kwq4d\""
Jul 28 22:47:29.764366 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:29.764240    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-9bcb6e\""
Jul 28 22:47:29.771698 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:29.771673    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9bcb6e-6df6f9fc64-jjh7q"]
Jul 28 22:47:29.791122 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:29.791099    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qnzt\" (UniqueName: \"kubernetes.io/projected/03d0f9b9-0aff-483a-8145-ba9d7bc64fef-kube-api-access-2qnzt\") pod \"payload-processing-e2e-api-b-9bcb6e-6df6f9fc64-jjh7q\" (UID: \"03d0f9b9-0aff-483a-8145-ba9d7bc64fef\") " pod="openshift-ingress/payload-processing-e2e-api-b-9bcb6e-6df6f9fc64-jjh7q"
Jul 28 22:47:29.791246 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:29.791146    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03d0f9b9-0aff-483a-8145-ba9d7bc64fef-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9bcb6e-6df6f9fc64-jjh7q\" (UID: \"03d0f9b9-0aff-483a-8145-ba9d7bc64fef\") " pod="openshift-ingress/payload-processing-e2e-api-b-9bcb6e-6df6f9fc64-jjh7q"
Jul 28 22:47:29.891669 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:29.891638    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03d0f9b9-0aff-483a-8145-ba9d7bc64fef-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9bcb6e-6df6f9fc64-jjh7q\" (UID: \"03d0f9b9-0aff-483a-8145-ba9d7bc64fef\") " pod="openshift-ingress/payload-processing-e2e-api-b-9bcb6e-6df6f9fc64-jjh7q"
Jul 28 22:47:29.891796 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:29.891781    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qnzt\" (UniqueName: \"kubernetes.io/projected/03d0f9b9-0aff-483a-8145-ba9d7bc64fef-kube-api-access-2qnzt\") pod \"payload-processing-e2e-api-b-9bcb6e-6df6f9fc64-jjh7q\" (UID: \"03d0f9b9-0aff-483a-8145-ba9d7bc64fef\") " pod="openshift-ingress/payload-processing-e2e-api-b-9bcb6e-6df6f9fc64-jjh7q"
Jul 28 22:47:29.892321 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:29.892301    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03d0f9b9-0aff-483a-8145-ba9d7bc64fef-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9bcb6e-6df6f9fc64-jjh7q\" (UID: \"03d0f9b9-0aff-483a-8145-ba9d7bc64fef\") " pod="openshift-ingress/payload-processing-e2e-api-b-9bcb6e-6df6f9fc64-jjh7q"
Jul 28 22:47:29.901042 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:29.901016    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qnzt\" (UniqueName: \"kubernetes.io/projected/03d0f9b9-0aff-483a-8145-ba9d7bc64fef-kube-api-access-2qnzt\") pod \"payload-processing-e2e-api-b-9bcb6e-6df6f9fc64-jjh7q\" (UID: \"03d0f9b9-0aff-483a-8145-ba9d7bc64fef\") " pod="openshift-ingress/payload-processing-e2e-api-b-9bcb6e-6df6f9fc64-jjh7q"
Jul 28 22:47:30.074762 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:30.074686    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9bcb6e-6df6f9fc64-jjh7q"
Jul 28 22:47:30.210638 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:30.210610    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9bcb6e-6df6f9fc64-jjh7q"]
Jul 28 22:47:30.212802 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:47:30.212758    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03d0f9b9_0aff_483a_8145_ba9d7bc64fef.slice/crio-168e8c56517ced56fa3b22d7b86abb96cf36ab177a9b99001878a36ee4758fe4 WatchSource:0}: Error finding container 168e8c56517ced56fa3b22d7b86abb96cf36ab177a9b99001878a36ee4758fe4: Status 404 returned error can't find the container with id 168e8c56517ced56fa3b22d7b86abb96cf36ab177a9b99001878a36ee4758fe4
Jul 28 22:47:30.446379 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:30.446324    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9bcb6e-5b5dbf7775-bcjqg" event={"ID":"dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90","Type":"ContainerStarted","Data":"d09802b6f9c19e331a5c8dacbd2f8c1b34860af19eadb8b881026f68e8e4b2df"}
Jul 28 22:47:30.446563 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:30.446402    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9bcb6e-5b5dbf7775-bcjqg"
Jul 28 22:47:30.446563 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:30.446426    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9bcb6e-5b5dbf7775-bcjqg" event={"ID":"dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90","Type":"ContainerStarted","Data":"47647d771980cb863906120fa0858642d9c1f166e5c5512ac4fad24863c393df"}
Jul 28 22:47:30.448008 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:30.447979    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9bcb6e-6df6f9fc64-jjh7q" event={"ID":"03d0f9b9-0aff-483a-8145-ba9d7bc64fef","Type":"ContainerStarted","Data":"3fefd89377cc322a2d3e496a372eb11d3ff6f062d50e97a5ca22d3aadbcc2019"}
Jul 28 22:47:30.448132 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:30.448030    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9bcb6e-6df6f9fc64-jjh7q" event={"ID":"03d0f9b9-0aff-483a-8145-ba9d7bc64fef","Type":"ContainerStarted","Data":"168e8c56517ced56fa3b22d7b86abb96cf36ab177a9b99001878a36ee4758fe4"}
Jul 28 22:47:30.448132 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:30.448052    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-9bcb6e-6df6f9fc64-jjh7q"
Jul 28 22:47:30.462594 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:30.462549    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9bcb6e-5b5dbf7775-bcjqg" podStartSLOduration=2.069218569 podStartE2EDuration="2.462532222s" totalImagesPullingTime="393.313653ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:47:30.462306781 +0000 UTC m=+2547.842444425" watchObservedRunningTime="2026-07-28 22:47:30.462532222 +0000 UTC m=+2547.842669869"
Jul 28 22:47:30.480192 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:30.480141    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-9bcb6e-6df6f9fc64-jjh7q" podStartSLOduration=1.480125296 podStartE2EDuration="1.480125296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:47:30.479316906 +0000 UTC m=+2547.859454561" watchObservedRunningTime="2026-07-28 22:47:30.480125296 +0000 UTC m=+2547.860263016"
Jul 28 22:47:36.458501 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:36.458468    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9bcb6e-5b5dbf7775-bcjqg"
Jul 28 22:47:41.455738 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:47:41.455703    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-9bcb6e-6df6f9fc64-jjh7q"
Jul 28 22:48:12.033140 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.033101    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-26e94d-665b4fdb49-tvprt"]
Jul 28 22:48:12.033758 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.033377    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26e94d-665b4fdb49-tvprt" podUID="6bbf6abb-c937-4ce9-8ba6-c02af67e6983" containerName="maas-api" containerID="cri-o://790d46358f833c3174b4515fdc540a166768883f4531130a03c8a504ce13aacc" gracePeriod=30
Jul 28 22:48:12.098199 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.098167    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-26e94d-5bb8f9dcb8-78zpb"]
Jul 28 22:48:12.098475 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.098429    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-26e94d-5bb8f9dcb8-78zpb" podUID="39aedb24-3827-4baf-84e3-6011dcf98e2b" containerName="payload-processing" containerID="cri-o://f9b281f4acc6ac5d84339e62ead9923064c3dda4347e4f80a6ceb4ef56d6d651" gracePeriod=30
Jul 28 22:48:12.106917 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.106882    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-26e94d-cbdd7fff6-hb7r9"]
Jul 28 22:48:12.107244 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.107211    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-26e94d-cbdd7fff6-hb7r9" podUID="d5a34a78-5df1-465c-9a98-adf40ab144d0" containerName="payload-pre-processing" containerID="cri-o://88cbd4669adeea3024979c407acbe22eb94d1ad4c4b1dbc892541ee71061e33c" gracePeriod=30
Jul 28 22:48:12.308628 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.308171    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-26e94d-cbdd7fff6-hb7r9" podUID="d5a34a78-5df1-465c-9a98-adf40ab144d0" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.68:9004: connect: connection refused"
Jul 28 22:48:12.308628 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.308239    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-a-26e94d-5bb8f9dcb8-78zpb" podUID="39aedb24-3827-4baf-84e3-6011dcf98e2b" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.69:9004: connect: connection refused"
Jul 28 22:48:12.376225 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.376197    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26e94d-665b4fdb49-tvprt"
Jul 28 22:48:12.413612 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.413584    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-26e94d-5bb8f9dcb8-78zpb"
Jul 28 22:48:12.422503 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.422480    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-26e94d-cbdd7fff6-hb7r9"
Jul 28 22:48:12.490406 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.490377    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bbf6abb-c937-4ce9-8ba6-c02af67e6983-openshift-service-ca\") pod \"6bbf6abb-c937-4ce9-8ba6-c02af67e6983\" (UID: \"6bbf6abb-c937-4ce9-8ba6-c02af67e6983\") "
Jul 28 22:48:12.490564 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.490418    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5a34a78-5df1-465c-9a98-adf40ab144d0-plugins-config-volume\") pod \"d5a34a78-5df1-465c-9a98-adf40ab144d0\" (UID: \"d5a34a78-5df1-465c-9a98-adf40ab144d0\") "
Jul 28 22:48:12.490564 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.490453    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2ld7\" (UniqueName: \"kubernetes.io/projected/d5a34a78-5df1-465c-9a98-adf40ab144d0-kube-api-access-b2ld7\") pod \"d5a34a78-5df1-465c-9a98-adf40ab144d0\" (UID: \"d5a34a78-5df1-465c-9a98-adf40ab144d0\") "
Jul 28 22:48:12.490564 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.490479    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bbx6\" (UniqueName: \"kubernetes.io/projected/39aedb24-3827-4baf-84e3-6011dcf98e2b-kube-api-access-2bbx6\") pod \"39aedb24-3827-4baf-84e3-6011dcf98e2b\" (UID: \"39aedb24-3827-4baf-84e3-6011dcf98e2b\") "
Jul 28 22:48:12.490564 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.490504    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39aedb24-3827-4baf-84e3-6011dcf98e2b-plugins-config-volume\") pod \"39aedb24-3827-4baf-84e3-6011dcf98e2b\" (UID: \"39aedb24-3827-4baf-84e3-6011dcf98e2b\") "
Jul 28 22:48:12.490564 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.490555    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bbf6abb-c937-4ce9-8ba6-c02af67e6983-maas-api-tls\") pod \"6bbf6abb-c937-4ce9-8ba6-c02af67e6983\" (UID: \"6bbf6abb-c937-4ce9-8ba6-c02af67e6983\") "
Jul 28 22:48:12.490838 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.490629    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9w8vw\" (UniqueName: \"kubernetes.io/projected/6bbf6abb-c937-4ce9-8ba6-c02af67e6983-kube-api-access-9w8vw\") pod \"6bbf6abb-c937-4ce9-8ba6-c02af67e6983\" (UID: \"6bbf6abb-c937-4ce9-8ba6-c02af67e6983\") "
Jul 28 22:48:12.490838 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.490815    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a34a78-5df1-465c-9a98-adf40ab144d0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d5a34a78-5df1-465c-9a98-adf40ab144d0" (UID: "d5a34a78-5df1-465c-9a98-adf40ab144d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:48:12.490962 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.490896    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5a34a78-5df1-465c-9a98-adf40ab144d0-plugins-config-volume\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:48:12.490962 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.490918    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39aedb24-3827-4baf-84e3-6011dcf98e2b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "39aedb24-3827-4baf-84e3-6011dcf98e2b" (UID: "39aedb24-3827-4baf-84e3-6011dcf98e2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:48:12.491079 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.491054    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bbf6abb-c937-4ce9-8ba6-c02af67e6983-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6bbf6abb-c937-4ce9-8ba6-c02af67e6983" (UID: "6bbf6abb-c937-4ce9-8ba6-c02af67e6983"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:48:12.492760 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.492729    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a34a78-5df1-465c-9a98-adf40ab144d0-kube-api-access-b2ld7" (OuterVolumeSpecName: "kube-api-access-b2ld7") pod "d5a34a78-5df1-465c-9a98-adf40ab144d0" (UID: "d5a34a78-5df1-465c-9a98-adf40ab144d0"). InnerVolumeSpecName "kube-api-access-b2ld7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:48:12.492760 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.492752    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39aedb24-3827-4baf-84e3-6011dcf98e2b-kube-api-access-2bbx6" (OuterVolumeSpecName: "kube-api-access-2bbx6") pod "39aedb24-3827-4baf-84e3-6011dcf98e2b" (UID: "39aedb24-3827-4baf-84e3-6011dcf98e2b"). InnerVolumeSpecName "kube-api-access-2bbx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:48:12.492904 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.492792    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bbf6abb-c937-4ce9-8ba6-c02af67e6983-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6bbf6abb-c937-4ce9-8ba6-c02af67e6983" (UID: "6bbf6abb-c937-4ce9-8ba6-c02af67e6983"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:48:12.492904 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.492799    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bbf6abb-c937-4ce9-8ba6-c02af67e6983-kube-api-access-9w8vw" (OuterVolumeSpecName: "kube-api-access-9w8vw") pod "6bbf6abb-c937-4ce9-8ba6-c02af67e6983" (UID: "6bbf6abb-c937-4ce9-8ba6-c02af67e6983"). InnerVolumeSpecName "kube-api-access-9w8vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:48:12.592016 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.591946    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w8vw\" (UniqueName: \"kubernetes.io/projected/6bbf6abb-c937-4ce9-8ba6-c02af67e6983-kube-api-access-9w8vw\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:48:12.592016 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.591970    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bbf6abb-c937-4ce9-8ba6-c02af67e6983-openshift-service-ca\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:48:12.592016 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.591980    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2ld7\" (UniqueName: \"kubernetes.io/projected/d5a34a78-5df1-465c-9a98-adf40ab144d0-kube-api-access-b2ld7\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:48:12.592016 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.591988    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bbx6\" (UniqueName: \"kubernetes.io/projected/39aedb24-3827-4baf-84e3-6011dcf98e2b-kube-api-access-2bbx6\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:48:12.592016 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.591996    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39aedb24-3827-4baf-84e3-6011dcf98e2b-plugins-config-volume\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:48:12.592016 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.592005    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bbf6abb-c937-4ce9-8ba6-c02af67e6983-maas-api-tls\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:48:12.657715 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.657683    2563 generic.go:356] "Generic (PLEG): container finished" podID="39aedb24-3827-4baf-84e3-6011dcf98e2b" containerID="f9b281f4acc6ac5d84339e62ead9923064c3dda4347e4f80a6ceb4ef56d6d651" exitCode=0
Jul 28 22:48:12.657875 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.657730    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-26e94d-5bb8f9dcb8-78zpb"
Jul 28 22:48:12.657875 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.657765    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-26e94d-5bb8f9dcb8-78zpb" event={"ID":"39aedb24-3827-4baf-84e3-6011dcf98e2b","Type":"ContainerDied","Data":"f9b281f4acc6ac5d84339e62ead9923064c3dda4347e4f80a6ceb4ef56d6d651"}
Jul 28 22:48:12.657875 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.657820    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-26e94d-5bb8f9dcb8-78zpb" event={"ID":"39aedb24-3827-4baf-84e3-6011dcf98e2b","Type":"ContainerDied","Data":"4974b5c6ad684ec52a5744ba545f9a912bdca864cfb4e82354f803de1d39279c"}
Jul 28 22:48:12.657875 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.657854    2563 scope.go:117] "RemoveContainer" containerID="f9b281f4acc6ac5d84339e62ead9923064c3dda4347e4f80a6ceb4ef56d6d651"
Jul 28 22:48:12.659174 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.659150    2563 generic.go:356] "Generic (PLEG): container finished" podID="6bbf6abb-c937-4ce9-8ba6-c02af67e6983" containerID="790d46358f833c3174b4515fdc540a166768883f4531130a03c8a504ce13aacc" exitCode=0
Jul 28 22:48:12.659312 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.659200    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26e94d-665b4fdb49-tvprt"
Jul 28 22:48:12.659312 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.659224    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26e94d-665b4fdb49-tvprt" event={"ID":"6bbf6abb-c937-4ce9-8ba6-c02af67e6983","Type":"ContainerDied","Data":"790d46358f833c3174b4515fdc540a166768883f4531130a03c8a504ce13aacc"}
Jul 28 22:48:12.659312 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.659266    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26e94d-665b4fdb49-tvprt" event={"ID":"6bbf6abb-c937-4ce9-8ba6-c02af67e6983","Type":"ContainerDied","Data":"8a9b0475847ddea3d9fe6b31652d471f754dd936f4d8abafecc23c4c8c6f3cfb"}
Jul 28 22:48:12.660687 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.660654    2563 generic.go:356] "Generic (PLEG): container finished" podID="d5a34a78-5df1-465c-9a98-adf40ab144d0" containerID="88cbd4669adeea3024979c407acbe22eb94d1ad4c4b1dbc892541ee71061e33c" exitCode=0
Jul 28 22:48:12.660802 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.660686    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-26e94d-cbdd7fff6-hb7r9" event={"ID":"d5a34a78-5df1-465c-9a98-adf40ab144d0","Type":"ContainerDied","Data":"88cbd4669adeea3024979c407acbe22eb94d1ad4c4b1dbc892541ee71061e33c"}
Jul 28 22:48:12.660802 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.660702    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-26e94d-cbdd7fff6-hb7r9"
Jul 28 22:48:12.660802 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.660741    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-26e94d-cbdd7fff6-hb7r9" event={"ID":"d5a34a78-5df1-465c-9a98-adf40ab144d0","Type":"ContainerDied","Data":"07eaf0d32b60660c93e2c7dee0fb28cb0936a79cc756890e31869943b34af0fc"}
Jul 28 22:48:12.669216 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.669200    2563 scope.go:117] "RemoveContainer" containerID="f9b281f4acc6ac5d84339e62ead9923064c3dda4347e4f80a6ceb4ef56d6d651"
Jul 28 22:48:12.669543 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:48:12.669515    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9b281f4acc6ac5d84339e62ead9923064c3dda4347e4f80a6ceb4ef56d6d651\": container with ID starting with f9b281f4acc6ac5d84339e62ead9923064c3dda4347e4f80a6ceb4ef56d6d651 not found: ID does not exist" containerID="f9b281f4acc6ac5d84339e62ead9923064c3dda4347e4f80a6ceb4ef56d6d651"
Jul 28 22:48:12.669639 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.669555    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9b281f4acc6ac5d84339e62ead9923064c3dda4347e4f80a6ceb4ef56d6d651"} err="failed to get container status \"f9b281f4acc6ac5d84339e62ead9923064c3dda4347e4f80a6ceb4ef56d6d651\": rpc error: code = NotFound desc = could not find container \"f9b281f4acc6ac5d84339e62ead9923064c3dda4347e4f80a6ceb4ef56d6d651\": container with ID starting with f9b281f4acc6ac5d84339e62ead9923064c3dda4347e4f80a6ceb4ef56d6d651 not found: ID does not exist"
Jul 28 22:48:12.669639 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.669580    2563 scope.go:117] "RemoveContainer" containerID="790d46358f833c3174b4515fdc540a166768883f4531130a03c8a504ce13aacc"
Jul 28 22:48:12.677803 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.677786    2563 scope.go:117] "RemoveContainer" containerID="790d46358f833c3174b4515fdc540a166768883f4531130a03c8a504ce13aacc"
Jul 28 22:48:12.678047 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:48:12.678027    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"790d46358f833c3174b4515fdc540a166768883f4531130a03c8a504ce13aacc\": container with ID starting with 790d46358f833c3174b4515fdc540a166768883f4531130a03c8a504ce13aacc not found: ID does not exist" containerID="790d46358f833c3174b4515fdc540a166768883f4531130a03c8a504ce13aacc"
Jul 28 22:48:12.678100 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.678054    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"790d46358f833c3174b4515fdc540a166768883f4531130a03c8a504ce13aacc"} err="failed to get container status \"790d46358f833c3174b4515fdc540a166768883f4531130a03c8a504ce13aacc\": rpc error: code = NotFound desc = could not find container \"790d46358f833c3174b4515fdc540a166768883f4531130a03c8a504ce13aacc\": container with ID starting with 790d46358f833c3174b4515fdc540a166768883f4531130a03c8a504ce13aacc not found: ID does not exist"
Jul 28 22:48:12.678100 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.678072    2563 scope.go:117] "RemoveContainer" containerID="88cbd4669adeea3024979c407acbe22eb94d1ad4c4b1dbc892541ee71061e33c"
Jul 28 22:48:12.681888 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.681868    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-26e94d-5bb8f9dcb8-78zpb"]
Jul 28 22:48:12.685377 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.685338    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-26e94d-5bb8f9dcb8-78zpb"]
Jul 28 22:48:12.686813 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.686798    2563 scope.go:117] "RemoveContainer" containerID="88cbd4669adeea3024979c407acbe22eb94d1ad4c4b1dbc892541ee71061e33c"
Jul 28 22:48:12.687055 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:48:12.687039    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88cbd4669adeea3024979c407acbe22eb94d1ad4c4b1dbc892541ee71061e33c\": container with ID starting with 88cbd4669adeea3024979c407acbe22eb94d1ad4c4b1dbc892541ee71061e33c not found: ID does not exist" containerID="88cbd4669adeea3024979c407acbe22eb94d1ad4c4b1dbc892541ee71061e33c"
Jul 28 22:48:12.687100 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.687063    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88cbd4669adeea3024979c407acbe22eb94d1ad4c4b1dbc892541ee71061e33c"} err="failed to get container status \"88cbd4669adeea3024979c407acbe22eb94d1ad4c4b1dbc892541ee71061e33c\": rpc error: code = NotFound desc = could not find container \"88cbd4669adeea3024979c407acbe22eb94d1ad4c4b1dbc892541ee71061e33c\": container with ID starting with 88cbd4669adeea3024979c407acbe22eb94d1ad4c4b1dbc892541ee71061e33c not found: ID does not exist"
Jul 28 22:48:12.695231 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.695210    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-26e94d-665b4fdb49-tvprt"]
Jul 28 22:48:12.700690 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.700672    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-26e94d-665b4fdb49-tvprt"]
Jul 28 22:48:12.710619 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.710599    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-26e94d-cbdd7fff6-hb7r9"]
Jul 28 22:48:12.714955 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:12.714931    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-26e94d-cbdd7fff6-hb7r9"]
Jul 28 22:48:13.130973 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:13.130940    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39aedb24-3827-4baf-84e3-6011dcf98e2b" path="/var/lib/kubelet/pods/39aedb24-3827-4baf-84e3-6011dcf98e2b/volumes"
Jul 28 22:48:13.131347 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:13.131334    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bbf6abb-c937-4ce9-8ba6-c02af67e6983" path="/var/lib/kubelet/pods/6bbf6abb-c937-4ce9-8ba6-c02af67e6983/volumes"
Jul 28 22:48:13.131662 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:13.131651    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5a34a78-5df1-465c-9a98-adf40ab144d0" path="/var/lib/kubelet/pods/d5a34a78-5df1-465c-9a98-adf40ab144d0/volumes"
Jul 28 22:48:15.309296 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:15.309234    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26e94d-665b4fdb49-tvprt" podUID="6bbf6abb-c937-4ce9-8ba6-c02af67e6983" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.67:8443/health\": context deadline exceeded"
Jul 28 22:48:34.692234 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:34.692146    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9bcb6e-5b5dbf7775-bcjqg"]
Jul 28 22:48:34.692688 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:34.692446    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9bcb6e-5b5dbf7775-bcjqg" podUID="dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90" containerName="maas-api" containerID="cri-o://d09802b6f9c19e331a5c8dacbd2f8c1b34860af19eadb8b881026f68e8e4b2df" gracePeriod=30
Jul 28 22:48:34.735027 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:34.734994    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9bcb6e-6df6f9fc64-jjh7q"]
Jul 28 22:48:34.735248 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:34.735221    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-9bcb6e-6df6f9fc64-jjh7q" podUID="03d0f9b9-0aff-483a-8145-ba9d7bc64fef" containerName="payload-processing" containerID="cri-o://3fefd89377cc322a2d3e496a372eb11d3ff6f062d50e97a5ca22d3aadbcc2019" gracePeriod=30
Jul 28 22:48:35.010215 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:35.010188    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9bcb6e-5b5dbf7775-bcjqg"
Jul 28 22:48:35.042700 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:35.042670    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9bcb6e-6df6f9fc64-jjh7q"
Jul 28 22:48:35.091334 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:35.091305    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90-maas-api-tls\") pod \"dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90\" (UID: \"dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90\") "
Jul 28 22:48:35.091517 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:35.091378    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lp8p8\" (UniqueName: \"kubernetes.io/projected/dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90-kube-api-access-lp8p8\") pod \"dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90\" (UID: \"dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90\") "
Jul 28 22:48:35.091517 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:35.091434    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90-openshift-service-ca\") pod \"dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90\" (UID: \"dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90\") "
Jul 28 22:48:35.091794 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:35.091771    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90" (UID: "dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:48:35.093301 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:35.093280    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90-kube-api-access-lp8p8" (OuterVolumeSpecName: "kube-api-access-lp8p8") pod "dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90" (UID: "dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90"). InnerVolumeSpecName "kube-api-access-lp8p8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:48:35.093398 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:35.093293    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90" (UID: "dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:48:35.192658 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:35.192632    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03d0f9b9-0aff-483a-8145-ba9d7bc64fef-plugins-config-volume\") pod \"03d0f9b9-0aff-483a-8145-ba9d7bc64fef\" (UID: \"03d0f9b9-0aff-483a-8145-ba9d7bc64fef\") "
Jul 28 22:48:35.192797 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:35.192679    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qnzt\" (UniqueName: \"kubernetes.io/projected/03d0f9b9-0aff-483a-8145-ba9d7bc64fef-kube-api-access-2qnzt\") pod \"03d0f9b9-0aff-483a-8145-ba9d7bc64fef\" (UID: \"03d0f9b9-0aff-483a-8145-ba9d7bc64fef\") "
Jul 28 22:48:35.192917 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:35.192902    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90-maas-api-tls\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:48:35.192966 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:35.192921    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp8p8\" (UniqueName: \"kubernetes.io/projected/dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90-kube-api-access-lp8p8\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:48:35.192966 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:35.192933    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90-openshift-service-ca\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:48:35.193034 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:35.192955    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03d0f9b9-0aff-483a-8145-ba9d7bc64fef-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "03d0f9b9-0aff-483a-8145-ba9d7bc64fef" (UID: "03d0f9b9-0aff-483a-8145-ba9d7bc64fef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 22:48:35.194581 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:35.194560    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03d0f9b9-0aff-483a-8145-ba9d7bc64fef-kube-api-access-2qnzt" (OuterVolumeSpecName: "kube-api-access-2qnzt") pod "03d0f9b9-0aff-483a-8145-ba9d7bc64fef" (UID: "03d0f9b9-0aff-483a-8145-ba9d7bc64fef"). InnerVolumeSpecName "kube-api-access-2qnzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:48:35.294274 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:35.294193    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03d0f9b9-0aff-483a-8145-ba9d7bc64fef-plugins-config-volume\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:48:35.294274 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:35.294221    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qnzt\" (UniqueName: \"kubernetes.io/projected/03d0f9b9-0aff-483a-8145-ba9d7bc64fef-kube-api-access-2qnzt\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:48:35.775656 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:35.775623    2563 generic.go:356] "Generic (PLEG): container finished" podID="dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90" containerID="d09802b6f9c19e331a5c8dacbd2f8c1b34860af19eadb8b881026f68e8e4b2df" exitCode=0
Jul 28 22:48:35.776073 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:35.775667    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9bcb6e-5b5dbf7775-bcjqg"
Jul 28 22:48:35.776073 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:35.775706    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9bcb6e-5b5dbf7775-bcjqg" event={"ID":"dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90","Type":"ContainerDied","Data":"d09802b6f9c19e331a5c8dacbd2f8c1b34860af19eadb8b881026f68e8e4b2df"}
Jul 28 22:48:35.776073 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:35.775764    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9bcb6e-5b5dbf7775-bcjqg" event={"ID":"dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90","Type":"ContainerDied","Data":"47647d771980cb863906120fa0858642d9c1f166e5c5512ac4fad24863c393df"}
Jul 28 22:48:35.776073 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:35.775801    2563 scope.go:117] "RemoveContainer" containerID="d09802b6f9c19e331a5c8dacbd2f8c1b34860af19eadb8b881026f68e8e4b2df"
Jul 28 22:48:35.777179 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:35.777096    2563 generic.go:356] "Generic (PLEG): container finished" podID="03d0f9b9-0aff-483a-8145-ba9d7bc64fef" containerID="3fefd89377cc322a2d3e496a372eb11d3ff6f062d50e97a5ca22d3aadbcc2019" exitCode=0
Jul 28 22:48:35.777179 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:35.777124    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9bcb6e-6df6f9fc64-jjh7q" event={"ID":"03d0f9b9-0aff-483a-8145-ba9d7bc64fef","Type":"ContainerDied","Data":"3fefd89377cc322a2d3e496a372eb11d3ff6f062d50e97a5ca22d3aadbcc2019"}
Jul 28 22:48:35.777179 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:35.777151    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9bcb6e-6df6f9fc64-jjh7q"
Jul 28 22:48:35.777179 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:35.777167    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9bcb6e-6df6f9fc64-jjh7q" event={"ID":"03d0f9b9-0aff-483a-8145-ba9d7bc64fef","Type":"ContainerDied","Data":"168e8c56517ced56fa3b22d7b86abb96cf36ab177a9b99001878a36ee4758fe4"}
Jul 28 22:48:35.784493 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:35.784476    2563 scope.go:117] "RemoveContainer" containerID="d09802b6f9c19e331a5c8dacbd2f8c1b34860af19eadb8b881026f68e8e4b2df"
Jul 28 22:48:35.784798 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:48:35.784766    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d09802b6f9c19e331a5c8dacbd2f8c1b34860af19eadb8b881026f68e8e4b2df\": container with ID starting with d09802b6f9c19e331a5c8dacbd2f8c1b34860af19eadb8b881026f68e8e4b2df not found: ID does not exist" containerID="d09802b6f9c19e331a5c8dacbd2f8c1b34860af19eadb8b881026f68e8e4b2df"
Jul 28 22:48:35.784888 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:35.784802    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d09802b6f9c19e331a5c8dacbd2f8c1b34860af19eadb8b881026f68e8e4b2df"} err="failed to get container status \"d09802b6f9c19e331a5c8dacbd2f8c1b34860af19eadb8b881026f68e8e4b2df\": rpc error: code = NotFound desc = could not find container \"d09802b6f9c19e331a5c8dacbd2f8c1b34860af19eadb8b881026f68e8e4b2df\": container with ID starting with d09802b6f9c19e331a5c8dacbd2f8c1b34860af19eadb8b881026f68e8e4b2df not found: ID does not exist"
Jul 28 22:48:35.784888 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:35.784826    2563 scope.go:117] "RemoveContainer" containerID="3fefd89377cc322a2d3e496a372eb11d3ff6f062d50e97a5ca22d3aadbcc2019"
Jul 28 22:48:35.792416 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:35.792400    2563 scope.go:117] "RemoveContainer" containerID="3fefd89377cc322a2d3e496a372eb11d3ff6f062d50e97a5ca22d3aadbcc2019"
Jul 28 22:48:35.792671 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:48:35.792653    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fefd89377cc322a2d3e496a372eb11d3ff6f062d50e97a5ca22d3aadbcc2019\": container with ID starting with 3fefd89377cc322a2d3e496a372eb11d3ff6f062d50e97a5ca22d3aadbcc2019 not found: ID does not exist" containerID="3fefd89377cc322a2d3e496a372eb11d3ff6f062d50e97a5ca22d3aadbcc2019"
Jul 28 22:48:35.792724 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:35.792679    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fefd89377cc322a2d3e496a372eb11d3ff6f062d50e97a5ca22d3aadbcc2019"} err="failed to get container status \"3fefd89377cc322a2d3e496a372eb11d3ff6f062d50e97a5ca22d3aadbcc2019\": rpc error: code = NotFound desc = could not find container \"3fefd89377cc322a2d3e496a372eb11d3ff6f062d50e97a5ca22d3aadbcc2019\": container with ID starting with 3fefd89377cc322a2d3e496a372eb11d3ff6f062d50e97a5ca22d3aadbcc2019 not found: ID does not exist"
Jul 28 22:48:35.798335 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:35.798307    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9bcb6e-5b5dbf7775-bcjqg"]
Jul 28 22:48:35.802490 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:35.802466    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9bcb6e-5b5dbf7775-bcjqg"]
Jul 28 22:48:35.815932 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:35.815906    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9bcb6e-6df6f9fc64-jjh7q"]
Jul 28 22:48:35.818740 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:35.818718    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9bcb6e-6df6f9fc64-jjh7q"]
Jul 28 22:48:37.135623 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:37.135592    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03d0f9b9-0aff-483a-8145-ba9d7bc64fef" path="/var/lib/kubelet/pods/03d0f9b9-0aff-483a-8145-ba9d7bc64fef/volumes"
Jul 28 22:48:37.136027 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:37.135976    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90" path="/var/lib/kubelet/pods/dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90/volumes"
Jul 28 22:48:44.965562 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:44.965523    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-3682ae-5f995b88d9-vclmw"]
Jul 28 22:48:44.966212 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:44.966046    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5a34a78-5df1-465c-9a98-adf40ab144d0" containerName="payload-pre-processing"
Jul 28 22:48:44.966212 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:44.966063    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5a34a78-5df1-465c-9a98-adf40ab144d0" containerName="payload-pre-processing"
Jul 28 22:48:44.966212 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:44.966078    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90" containerName="maas-api"
Jul 28 22:48:44.966212 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:44.966087    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90" containerName="maas-api"
Jul 28 22:48:44.966212 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:44.966116    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bbf6abb-c937-4ce9-8ba6-c02af67e6983" containerName="maas-api"
Jul 28 22:48:44.966212 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:44.966126    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bbf6abb-c937-4ce9-8ba6-c02af67e6983" containerName="maas-api"
Jul 28 22:48:44.966212 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:44.966139    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03d0f9b9-0aff-483a-8145-ba9d7bc64fef" containerName="payload-processing"
Jul 28 22:48:44.966212 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:44.966148    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="03d0f9b9-0aff-483a-8145-ba9d7bc64fef" containerName="payload-processing"
Jul 28 22:48:44.966212 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:44.966166    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39aedb24-3827-4baf-84e3-6011dcf98e2b" containerName="payload-processing"
Jul 28 22:48:44.966212 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:44.966176    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="39aedb24-3827-4baf-84e3-6011dcf98e2b" containerName="payload-processing"
Jul 28 22:48:44.966779 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:44.966254    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="03d0f9b9-0aff-483a-8145-ba9d7bc64fef" containerName="payload-processing"
Jul 28 22:48:44.966779 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:44.966268    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bbf6abb-c937-4ce9-8ba6-c02af67e6983" containerName="maas-api"
Jul 28 22:48:44.966779 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:44.966279    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="39aedb24-3827-4baf-84e3-6011dcf98e2b" containerName="payload-processing"
Jul 28 22:48:44.966779 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:44.966290    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5a34a78-5df1-465c-9a98-adf40ab144d0" containerName="payload-pre-processing"
Jul 28 22:48:44.966779 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:44.966299    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc2a2ccc-ae34-4bb5-bfa7-3e0a5fbe5e90" containerName="maas-api"
Jul 28 22:48:44.969755 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:44.969734    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3682ae-5f995b88d9-vclmw"
Jul 28 22:48:44.972563 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:44.972536    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-3682ae-serving-cert\""
Jul 28 22:48:44.975165 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:44.975136    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a2a293b-ce61-4e53-8821-79c92242ae07-maas-api-tls\") pod \"maas-api-e2e-shared-a-3682ae-5f995b88d9-vclmw\" (UID: \"0a2a293b-ce61-4e53-8821-79c92242ae07\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3682ae-5f995b88d9-vclmw"
Jul 28 22:48:44.975287 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:44.975207    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt7kg\" (UniqueName: \"kubernetes.io/projected/0a2a293b-ce61-4e53-8821-79c92242ae07-kube-api-access-dt7kg\") pod \"maas-api-e2e-shared-a-3682ae-5f995b88d9-vclmw\" (UID: \"0a2a293b-ce61-4e53-8821-79c92242ae07\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3682ae-5f995b88d9-vclmw"
Jul 28 22:48:44.975287 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:44.975226    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a2a293b-ce61-4e53-8821-79c92242ae07-openshift-service-ca\") pod \"maas-api-e2e-shared-a-3682ae-5f995b88d9-vclmw\" (UID: \"0a2a293b-ce61-4e53-8821-79c92242ae07\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3682ae-5f995b88d9-vclmw"
Jul 28 22:48:44.983920 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:44.983894    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-3682ae-5f995b88d9-vclmw"]
Jul 28 22:48:45.075868 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:45.075830    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a2a293b-ce61-4e53-8821-79c92242ae07-maas-api-tls\") pod \"maas-api-e2e-shared-a-3682ae-5f995b88d9-vclmw\" (UID: \"0a2a293b-ce61-4e53-8821-79c92242ae07\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3682ae-5f995b88d9-vclmw"
Jul 28 22:48:45.076038 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:45.075895    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt7kg\" (UniqueName: \"kubernetes.io/projected/0a2a293b-ce61-4e53-8821-79c92242ae07-kube-api-access-dt7kg\") pod \"maas-api-e2e-shared-a-3682ae-5f995b88d9-vclmw\" (UID: \"0a2a293b-ce61-4e53-8821-79c92242ae07\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3682ae-5f995b88d9-vclmw"
Jul 28 22:48:45.076038 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:45.075913    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a2a293b-ce61-4e53-8821-79c92242ae07-openshift-service-ca\") pod \"maas-api-e2e-shared-a-3682ae-5f995b88d9-vclmw\" (UID: \"0a2a293b-ce61-4e53-8821-79c92242ae07\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3682ae-5f995b88d9-vclmw"
Jul 28 22:48:45.076117 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:48:45.076078    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-3682ae-serving-cert: secret "maas-api-e2e-shared-a-3682ae-serving-cert" not found
Jul 28 22:48:45.076194 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:48:45.076181    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a2a293b-ce61-4e53-8821-79c92242ae07-maas-api-tls podName:0a2a293b-ce61-4e53-8821-79c92242ae07 nodeName:}" failed. No retries permitted until 2026-07-28 22:48:45.576161506 +0000 UTC m=+2622.956299134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0a2a293b-ce61-4e53-8821-79c92242ae07-maas-api-tls") pod "maas-api-e2e-shared-a-3682ae-5f995b88d9-vclmw" (UID: "0a2a293b-ce61-4e53-8821-79c92242ae07") : secret "maas-api-e2e-shared-a-3682ae-serving-cert" not found
Jul 28 22:48:45.076570 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:45.076552    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a2a293b-ce61-4e53-8821-79c92242ae07-openshift-service-ca\") pod \"maas-api-e2e-shared-a-3682ae-5f995b88d9-vclmw\" (UID: \"0a2a293b-ce61-4e53-8821-79c92242ae07\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3682ae-5f995b88d9-vclmw"
Jul 28 22:48:45.091374 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:45.091325    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt7kg\" (UniqueName: \"kubernetes.io/projected/0a2a293b-ce61-4e53-8821-79c92242ae07-kube-api-access-dt7kg\") pod \"maas-api-e2e-shared-a-3682ae-5f995b88d9-vclmw\" (UID: \"0a2a293b-ce61-4e53-8821-79c92242ae07\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3682ae-5f995b88d9-vclmw"
Jul 28 22:48:45.586569 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:45.586534    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a2a293b-ce61-4e53-8821-79c92242ae07-maas-api-tls\") pod \"maas-api-e2e-shared-a-3682ae-5f995b88d9-vclmw\" (UID: \"0a2a293b-ce61-4e53-8821-79c92242ae07\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3682ae-5f995b88d9-vclmw"
Jul 28 22:48:45.588972 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:45.588946    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a2a293b-ce61-4e53-8821-79c92242ae07-maas-api-tls\") pod \"maas-api-e2e-shared-a-3682ae-5f995b88d9-vclmw\" (UID: \"0a2a293b-ce61-4e53-8821-79c92242ae07\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3682ae-5f995b88d9-vclmw"
Jul 28 22:48:45.856899 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:45.856801    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-3682ae-7985768976-vtr77"]
Jul 28 22:48:45.863413 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:45.863389    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-3682ae-7985768976-vtr77"
Jul 28 22:48:45.866259 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:45.866236    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-3682ae\""
Jul 28 22:48:45.866419 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:45.866270    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-3682ae-dockercfg-fnbnz\""
Jul 28 22:48:45.870120 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:45.870097    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-3682ae-7985768976-vtr77"]
Jul 28 22:48:45.883073 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:45.883048    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3682ae-5f995b88d9-vclmw"
Jul 28 22:48:45.888859 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:45.888834    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acc7d7c1-cb50-4705-9676-3fedeacdda8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-3682ae-7985768976-vtr77\" (UID: \"acc7d7c1-cb50-4705-9676-3fedeacdda8c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-3682ae-7985768976-vtr77"
Jul 28 22:48:45.888992 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:45.888969    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9lzl\" (UniqueName: \"kubernetes.io/projected/acc7d7c1-cb50-4705-9676-3fedeacdda8c-kube-api-access-j9lzl\") pod \"payload-pre-processing-e2e-shared-a-3682ae-7985768976-vtr77\" (UID: \"acc7d7c1-cb50-4705-9676-3fedeacdda8c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-3682ae-7985768976-vtr77"
Jul 28 22:48:45.897508 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:45.897487    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-3682ae-68d4966958-dnpv4"]
Jul 28 22:48:45.900917 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:45.900899    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-3682ae-68d4966958-dnpv4"
Jul 28 22:48:45.911440 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:45.911347    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-3682ae-68d4966958-dnpv4"]
Jul 28 22:48:45.990032 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:45.989998    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9lzl\" (UniqueName: \"kubernetes.io/projected/acc7d7c1-cb50-4705-9676-3fedeacdda8c-kube-api-access-j9lzl\") pod \"payload-pre-processing-e2e-shared-a-3682ae-7985768976-vtr77\" (UID: \"acc7d7c1-cb50-4705-9676-3fedeacdda8c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-3682ae-7985768976-vtr77"
Jul 28 22:48:45.990516 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:45.990044    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shcnr\" (UniqueName: \"kubernetes.io/projected/7480613f-9415-433c-b63c-72501451864d-kube-api-access-shcnr\") pod \"payload-processing-e2e-shared-a-3682ae-68d4966958-dnpv4\" (UID: \"7480613f-9415-433c-b63c-72501451864d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-3682ae-68d4966958-dnpv4"
Jul 28 22:48:45.990516 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:45.990095    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7480613f-9415-433c-b63c-72501451864d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-3682ae-68d4966958-dnpv4\" (UID: \"7480613f-9415-433c-b63c-72501451864d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-3682ae-68d4966958-dnpv4"
Jul 28 22:48:45.990516 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:45.990147    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acc7d7c1-cb50-4705-9676-3fedeacdda8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-3682ae-7985768976-vtr77\" (UID: \"acc7d7c1-cb50-4705-9676-3fedeacdda8c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-3682ae-7985768976-vtr77"
Jul 28 22:48:45.991011 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:45.990986    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acc7d7c1-cb50-4705-9676-3fedeacdda8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-3682ae-7985768976-vtr77\" (UID: \"acc7d7c1-cb50-4705-9676-3fedeacdda8c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-3682ae-7985768976-vtr77"
Jul 28 22:48:46.001323 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:46.001212    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9lzl\" (UniqueName: \"kubernetes.io/projected/acc7d7c1-cb50-4705-9676-3fedeacdda8c-kube-api-access-j9lzl\") pod \"payload-pre-processing-e2e-shared-a-3682ae-7985768976-vtr77\" (UID: \"acc7d7c1-cb50-4705-9676-3fedeacdda8c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-3682ae-7985768976-vtr77"
Jul 28 22:48:46.028936 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:46.028904    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-3682ae-5f995b88d9-vclmw"]
Jul 28 22:48:46.032527 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:48:46.032501    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a2a293b_ce61_4e53_8821_79c92242ae07.slice/crio-b925da4721e9c32f6073103a9ecf62d4d3174faac187b182ad02b4397244f978 WatchSource:0}: Error finding container b925da4721e9c32f6073103a9ecf62d4d3174faac187b182ad02b4397244f978: Status 404 returned error can't find the container with id b925da4721e9c32f6073103a9ecf62d4d3174faac187b182ad02b4397244f978
Jul 28 22:48:46.091113 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:46.091084    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7480613f-9415-433c-b63c-72501451864d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-3682ae-68d4966958-dnpv4\" (UID: \"7480613f-9415-433c-b63c-72501451864d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-3682ae-68d4966958-dnpv4"
Jul 28 22:48:46.091296 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:46.091211    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shcnr\" (UniqueName: \"kubernetes.io/projected/7480613f-9415-433c-b63c-72501451864d-kube-api-access-shcnr\") pod \"payload-processing-e2e-shared-a-3682ae-68d4966958-dnpv4\" (UID: \"7480613f-9415-433c-b63c-72501451864d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-3682ae-68d4966958-dnpv4"
Jul 28 22:48:46.091993 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:46.091969    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7480613f-9415-433c-b63c-72501451864d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-3682ae-68d4966958-dnpv4\" (UID: \"7480613f-9415-433c-b63c-72501451864d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-3682ae-68d4966958-dnpv4"
Jul 28 22:48:46.099901 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:46.099877    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shcnr\" (UniqueName: \"kubernetes.io/projected/7480613f-9415-433c-b63c-72501451864d-kube-api-access-shcnr\") pod \"payload-processing-e2e-shared-a-3682ae-68d4966958-dnpv4\" (UID: \"7480613f-9415-433c-b63c-72501451864d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-3682ae-68d4966958-dnpv4"
Jul 28 22:48:46.175402 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:46.175306    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-3682ae-7985768976-vtr77"
Jul 28 22:48:46.223119 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:46.223089    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-3682ae-68d4966958-dnpv4"
Jul 28 22:48:46.367498 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:46.367472    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-3682ae-68d4966958-dnpv4"]
Jul 28 22:48:46.369419 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:48:46.369391    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7480613f_9415_433c_b63c_72501451864d.slice/crio-5b22f5ea6beb90708103eaf15de1d2e993c7415902802d75963fb749c647717f WatchSource:0}: Error finding container 5b22f5ea6beb90708103eaf15de1d2e993c7415902802d75963fb749c647717f: Status 404 returned error can't find the container with id 5b22f5ea6beb90708103eaf15de1d2e993c7415902802d75963fb749c647717f
Jul 28 22:48:46.526474 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:46.526426    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-3682ae-7985768976-vtr77"]
Jul 28 22:48:46.528662 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:48:46.528635    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc7d7c1_cb50_4705_9676_3fedeacdda8c.slice/crio-6fcac7381c8ca55ce97dafd6f558b4b6f38eac43e5cfd9734370c02b7da049eb WatchSource:0}: Error finding container 6fcac7381c8ca55ce97dafd6f558b4b6f38eac43e5cfd9734370c02b7da049eb: Status 404 returned error can't find the container with id 6fcac7381c8ca55ce97dafd6f558b4b6f38eac43e5cfd9734370c02b7da049eb
Jul 28 22:48:46.831974 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:46.831922    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-3682ae-7985768976-vtr77" event={"ID":"acc7d7c1-cb50-4705-9676-3fedeacdda8c","Type":"ContainerStarted","Data":"6f4df81472038ccee17dbd75e1b258d10f5bf3ca26fe17a6ce93f1c1aa717a12"}
Jul 28 22:48:46.832190 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:46.831987    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-3682ae-7985768976-vtr77"
Jul 28 22:48:46.832190 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:46.832008    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-3682ae-7985768976-vtr77" event={"ID":"acc7d7c1-cb50-4705-9676-3fedeacdda8c","Type":"ContainerStarted","Data":"6fcac7381c8ca55ce97dafd6f558b4b6f38eac43e5cfd9734370c02b7da049eb"}
Jul 28 22:48:46.834121 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:46.833736    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-3682ae-68d4966958-dnpv4" event={"ID":"7480613f-9415-433c-b63c-72501451864d","Type":"ContainerStarted","Data":"84b33b2e7c0da322da02d7fdcff6390ae5994b9d6c8de35ddee4053951d3070d"}
Jul 28 22:48:46.834121 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:46.833788    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-3682ae-68d4966958-dnpv4" event={"ID":"7480613f-9415-433c-b63c-72501451864d","Type":"ContainerStarted","Data":"5b22f5ea6beb90708103eaf15de1d2e993c7415902802d75963fb749c647717f"}
Jul 28 22:48:46.834121 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:46.833998    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-3682ae-68d4966958-dnpv4"
Jul 28 22:48:46.836030 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:46.836000    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3682ae-5f995b88d9-vclmw" event={"ID":"0a2a293b-ce61-4e53-8821-79c92242ae07","Type":"ContainerStarted","Data":"b73573f908364ba30d1db57bf2e07abcfb79d63a0a2648aba9df64643023eb75"}
Jul 28 22:48:46.836182 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:46.836048    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3682ae-5f995b88d9-vclmw" event={"ID":"0a2a293b-ce61-4e53-8821-79c92242ae07","Type":"ContainerStarted","Data":"b925da4721e9c32f6073103a9ecf62d4d3174faac187b182ad02b4397244f978"}
Jul 28 22:48:46.836182 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:46.836153    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3682ae-5f995b88d9-vclmw"
Jul 28 22:48:46.850696 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:46.850640    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-3682ae-7985768976-vtr77" podStartSLOduration=1.8506194329999999 podStartE2EDuration="1.850619433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:48:46.848768146 +0000 UTC m=+2624.228905817" watchObservedRunningTime="2026-07-28 22:48:46.850619433 +0000 UTC m=+2624.230757078"
Jul 28 22:48:46.866521 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:46.866471    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3682ae-5f995b88d9-vclmw" podStartSLOduration=2.488129222 podStartE2EDuration="2.866455787s" totalImagesPullingTime="378.326565ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:48:46.864538211 +0000 UTC m=+2624.244675858" watchObservedRunningTime="2026-07-28 22:48:46.866455787 +0000 UTC m=+2624.246593470"
Jul 28 22:48:46.881639 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:46.881591    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-3682ae-68d4966958-dnpv4" podStartSLOduration=1.881580174 podStartE2EDuration="1.881580174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:48:46.880411629 +0000 UTC m=+2624.260549273" watchObservedRunningTime="2026-07-28 22:48:46.881580174 +0000 UTC m=+2624.261717855"
Jul 28 22:48:52.848014 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:52.847985    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3682ae-5f995b88d9-vclmw"
Jul 28 22:48:57.844451 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:57.844417    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-3682ae-7985768976-vtr77"
Jul 28 22:48:57.844925 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:48:57.844509    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-3682ae-68d4966958-dnpv4"
Jul 28 22:49:13.101331 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:49:13.101294    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-aa2de7-6d886fb56c-ggntt"]
Jul 28 22:49:13.105401 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:49:13.105381    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-aa2de7-6d886fb56c-ggntt"
Jul 28 22:49:13.108853 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:49:13.108829    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-aa2de7-serving-cert\""
Jul 28 22:49:13.116977 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:49:13.116954    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-aa2de7-6d886fb56c-ggntt"]
Jul 28 22:49:13.227444 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:49:13.227403    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0668771a-becd-4b6c-b38e-36b46862a525-maas-api-tls\") pod \"maas-api-e2e-shared-b-aa2de7-6d886fb56c-ggntt\" (UID: \"0668771a-becd-4b6c-b38e-36b46862a525\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-aa2de7-6d886fb56c-ggntt"
Jul 28 22:49:13.227649 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:49:13.227557    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0668771a-becd-4b6c-b38e-36b46862a525-openshift-service-ca\") pod \"maas-api-e2e-shared-b-aa2de7-6d886fb56c-ggntt\" (UID: \"0668771a-becd-4b6c-b38e-36b46862a525\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-aa2de7-6d886fb56c-ggntt"
Jul 28 22:49:13.227649 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:49:13.227628    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqkpq\" (UniqueName: \"kubernetes.io/projected/0668771a-becd-4b6c-b38e-36b46862a525-kube-api-access-mqkpq\") pod \"maas-api-e2e-shared-b-aa2de7-6d886fb56c-ggntt\" (UID: \"0668771a-becd-4b6c-b38e-36b46862a525\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-aa2de7-6d886fb56c-ggntt"
Jul 28 22:49:13.328752 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:49:13.328719    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0668771a-becd-4b6c-b38e-36b46862a525-maas-api-tls\") pod \"maas-api-e2e-shared-b-aa2de7-6d886fb56c-ggntt\" (UID: \"0668771a-becd-4b6c-b38e-36b46862a525\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-aa2de7-6d886fb56c-ggntt"
Jul 28 22:49:13.328752 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:49:13.328753    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0668771a-becd-4b6c-b38e-36b46862a525-openshift-service-ca\") pod \"maas-api-e2e-shared-b-aa2de7-6d886fb56c-ggntt\" (UID: \"0668771a-becd-4b6c-b38e-36b46862a525\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-aa2de7-6d886fb56c-ggntt"
Jul 28 22:49:13.328966 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:49:13.328775    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqkpq\" (UniqueName: \"kubernetes.io/projected/0668771a-becd-4b6c-b38e-36b46862a525-kube-api-access-mqkpq\") pod \"maas-api-e2e-shared-b-aa2de7-6d886fb56c-ggntt\" (UID: \"0668771a-becd-4b6c-b38e-36b46862a525\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-aa2de7-6d886fb56c-ggntt"
Jul 28 22:49:13.329497 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:49:13.329477    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0668771a-becd-4b6c-b38e-36b46862a525-openshift-service-ca\") pod \"maas-api-e2e-shared-b-aa2de7-6d886fb56c-ggntt\" (UID: \"0668771a-becd-4b6c-b38e-36b46862a525\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-aa2de7-6d886fb56c-ggntt"
Jul 28 22:49:13.331151 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:49:13.331132    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0668771a-becd-4b6c-b38e-36b46862a525-maas-api-tls\") pod \"maas-api-e2e-shared-b-aa2de7-6d886fb56c-ggntt\" (UID: \"0668771a-becd-4b6c-b38e-36b46862a525\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-aa2de7-6d886fb56c-ggntt"
Jul 28 22:49:13.339091 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:49:13.339067    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqkpq\" (UniqueName: \"kubernetes.io/projected/0668771a-becd-4b6c-b38e-36b46862a525-kube-api-access-mqkpq\") pod \"maas-api-e2e-shared-b-aa2de7-6d886fb56c-ggntt\" (UID: \"0668771a-becd-4b6c-b38e-36b46862a525\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-aa2de7-6d886fb56c-ggntt"
Jul 28 22:49:13.416714 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:49:13.416629    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-aa2de7-6d886fb56c-ggntt"
Jul 28 22:49:13.550681 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:49:13.550654    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-aa2de7-6d886fb56c-ggntt"]
Jul 28 22:49:13.552786 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:49:13.552760    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0668771a_becd_4b6c_b38e_36b46862a525.slice/crio-64e903ea71eb53bea4db9514bfe553539059608ce6a21461af9f9caebc1b5ef2 WatchSource:0}: Error finding container 64e903ea71eb53bea4db9514bfe553539059608ce6a21461af9f9caebc1b5ef2: Status 404 returned error can't find the container with id 64e903ea71eb53bea4db9514bfe553539059608ce6a21461af9f9caebc1b5ef2
Jul 28 22:49:13.815538 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:49:13.815504    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-aa2de7-7f7cb8bfb-sr6jt"]
Jul 28 22:49:13.819754 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:49:13.819738    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-aa2de7-7f7cb8bfb-sr6jt"
Jul 28 22:49:13.822340 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:49:13.822316    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-aa2de7-dockercfg-dkd6c\""
Jul 28 22:49:13.822458 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:49:13.822321    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-aa2de7\""
Jul 28 22:49:13.832546 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:49:13.832519    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-aa2de7-7f7cb8bfb-sr6jt"]
Jul 28 22:49:13.934897 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:49:13.934857    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0242d98-9729-4007-9293-c893ec1a40fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-aa2de7-7f7cb8bfb-sr6jt\" (UID: \"c0242d98-9729-4007-9293-c893ec1a40fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-aa2de7-7f7cb8bfb-sr6jt"
Jul 28 22:49:13.935102 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:49:13.934913    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9whpx\" (UniqueName: \"kubernetes.io/projected/c0242d98-9729-4007-9293-c893ec1a40fc-kube-api-access-9whpx\") pod \"payload-pre-processing-e2e-shared-b-aa2de7-7f7cb8bfb-sr6jt\" (UID: \"c0242d98-9729-4007-9293-c893ec1a40fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-aa2de7-7f7cb8bfb-sr6jt"
Jul 28 22:49:13.974252 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:49:13.974221    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-aa2de7-6d886fb56c-ggntt" event={"ID":"0668771a-becd-4b6c-b38e-36b46862a525","Type":"ContainerStarted","Data":"64e903ea71eb53bea4db9514bfe553539059608ce6a21461af9f9caebc1b5ef2"}
Jul 28 22:49:14.035979 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:49:14.035941    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0242d98-9729-4007-9293-c893ec1a40fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-aa2de7-7f7cb8bfb-sr6jt\" (UID: \"c0242d98-9729-4007-9293-c893ec1a40fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-aa2de7-7f7cb8bfb-sr6jt"
Jul 28 22:49:14.035979 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:49:14.035991    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9whpx\" (UniqueName: \"kubernetes.io/projected/c0242d98-9729-4007-9293-c893ec1a40fc-kube-api-access-9whpx\") pod \"payload-pre-processing-e2e-shared-b-aa2de7-7f7cb8bfb-sr6jt\" (UID: \"c0242d98-9729-4007-9293-c893ec1a40fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-aa2de7-7f7cb8bfb-sr6jt"
Jul 28 22:49:14.036758 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:49:14.036737    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0242d98-9729-4007-9293-c893ec1a40fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-aa2de7-7f7cb8bfb-sr6jt\" (UID: \"c0242d98-9729-4007-9293-c893ec1a40fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-aa2de7-7f7cb8bfb-sr6jt"
Jul 28 22:49:14.044148 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:49:14.044125    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9whpx\" (UniqueName: \"kubernetes.io/projected/c0242d98-9729-4007-9293-c893ec1a40fc-kube-api-access-9whpx\") pod \"payload-pre-processing-e2e-shared-b-aa2de7-7f7cb8bfb-sr6jt\" (UID: \"c0242d98-9729-4007-9293-c893ec1a40fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-aa2de7-7f7cb8bfb-sr6jt"
Jul 28 22:49:14.131122 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:49:14.131035    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-aa2de7-7f7cb8bfb-sr6jt"
Jul 28 22:49:14.277406 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:49:14.277307    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-aa2de7-7f7cb8bfb-sr6jt"]
Jul 28 22:49:14.279965 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:49:14.279942    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0242d98_9729_4007_9293_c893ec1a40fc.slice/crio-239d1c13e5198cbb6b60cb8aa4b66c1763a9736da4040be5a5b53ed72774d5e1 WatchSource:0}: Error finding container 239d1c13e5198cbb6b60cb8aa4b66c1763a9736da4040be5a5b53ed72774d5e1: Status 404 returned error can't find the container with id 239d1c13e5198cbb6b60cb8aa4b66c1763a9736da4040be5a5b53ed72774d5e1
Jul 28 22:49:14.981127 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:49:14.981088    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-aa2de7-7f7cb8bfb-sr6jt" event={"ID":"c0242d98-9729-4007-9293-c893ec1a40fc","Type":"ContainerStarted","Data":"383f16c71200b070bd039b910a399267ae97ae974184082955d4779f8019c8be"}
Jul 28 22:49:14.981403 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:49:14.981152    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-aa2de7-7f7cb8bfb-sr6jt" event={"ID":"c0242d98-9729-4007-9293-c893ec1a40fc","Type":"ContainerStarted","Data":"239d1c13e5198cbb6b60cb8aa4b66c1763a9736da4040be5a5b53ed72774d5e1"}
Jul 28 22:49:14.981403 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:49:14.981250    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-aa2de7-7f7cb8bfb-sr6jt"
Jul 28 22:49:14.982955 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:49:14.982926    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-aa2de7-6d886fb56c-ggntt" event={"ID":"0668771a-becd-4b6c-b38e-36b46862a525","Type":"ContainerStarted","Data":"192e1bcf1a7a8aa4ec01015430210d7b7d73696ef05977de651965ac2a5e1981"}
Jul 28 22:49:14.983104 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:49:14.983081    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-aa2de7-6d886fb56c-ggntt"
Jul 28 22:49:14.997214 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:49:14.997172    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-aa2de7-7f7cb8bfb-sr6jt" podStartSLOduration=1.997158694 podStartE2EDuration="1.997158694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:49:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:49:14.996091843 +0000 UTC m=+2652.376229487" watchObservedRunningTime="2026-07-28 22:49:14.997158694 +0000 UTC m=+2652.377296338"
Jul 28 22:49:15.017219 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:49:15.017171    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-aa2de7-6d886fb56c-ggntt" podStartSLOduration=1.342293368 podStartE2EDuration="2.017155313s" totalImagesPullingTime="674.861945ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:49:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:49:15.016895521 +0000 UTC m=+2652.397033166" watchObservedRunningTime="2026-07-28 22:49:15.017155313 +0000 UTC m=+2652.397292958"
Jul 28 22:49:20.992887 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:49:20.992858    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-aa2de7-6d886fb56c-ggntt"
Jul 28 22:49:25.990120 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:49:25.990091    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-aa2de7-7f7cb8bfb-sr6jt"
Jul 28 22:50:03.214059 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:50:03.213990    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mcj8p_7bab32b0-13c6-4055-916e-a7454bafb5b6/ovn-acl-logging/0.log"
Jul 28 22:50:03.218707 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:50:03.218676    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mcj8p_7bab32b0-13c6-4055-916e-a7454bafb5b6/ovn-acl-logging/0.log"
Jul 28 22:54:38.266224 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:54:38.266180    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-3682ae/sub-test-model-3682ae-kserve-55b8499769-w8j84"]
Jul 28 22:54:38.270256 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:54:38.270237    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3682ae/sub-test-model-3682ae-kserve-55b8499769-w8j84"
Jul 28 22:54:38.273929 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:54:38.273900    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-3682ae\"/\"default-dockercfg-pn4x8\""
Jul 28 22:54:38.273929 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:54:38.273918    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-3682ae\"/\"openshift-service-ca.crt\""
Jul 28 22:54:38.274124 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:54:38.273922    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-3682ae\"/\"kube-root-ca.crt\""
Jul 28 22:54:38.274124 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:54:38.273920    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-3682ae\"/\"sub-test-model-3682ae-kserve-self-signed-certs\""
Jul 28 22:54:38.280723 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:54:38.279264    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-3682ae/sub-test-model-3682ae-kserve-55b8499769-w8j84"]
Jul 28 22:54:38.339870 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:54:38.339835    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb7435c6-b83a-479b-85aa-37264666c7a4-home\") pod \"sub-test-model-3682ae-kserve-55b8499769-w8j84\" (UID: \"bb7435c6-b83a-479b-85aa-37264666c7a4\") " pod="ai-tenant-e2e-shared-a-3682ae/sub-test-model-3682ae-kserve-55b8499769-w8j84"
Jul 28 22:54:38.340043 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:54:38.339899    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb7435c6-b83a-479b-85aa-37264666c7a4-dshm\") pod \"sub-test-model-3682ae-kserve-55b8499769-w8j84\" (UID: \"bb7435c6-b83a-479b-85aa-37264666c7a4\") " pod="ai-tenant-e2e-shared-a-3682ae/sub-test-model-3682ae-kserve-55b8499769-w8j84"
Jul 28 22:54:38.340043 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:54:38.339984    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb7435c6-b83a-479b-85aa-37264666c7a4-tmp-dir\") pod \"sub-test-model-3682ae-kserve-55b8499769-w8j84\" (UID: \"bb7435c6-b83a-479b-85aa-37264666c7a4\") " pod="ai-tenant-e2e-shared-a-3682ae/sub-test-model-3682ae-kserve-55b8499769-w8j84"
Jul 28 22:54:38.340043 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:54:38.340025    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbqc2\" (UniqueName: \"kubernetes.io/projected/bb7435c6-b83a-479b-85aa-37264666c7a4-kube-api-access-mbqc2\") pod \"sub-test-model-3682ae-kserve-55b8499769-w8j84\" (UID: \"bb7435c6-b83a-479b-85aa-37264666c7a4\") " pod="ai-tenant-e2e-shared-a-3682ae/sub-test-model-3682ae-kserve-55b8499769-w8j84"
Jul 28 22:54:38.340174 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:54:38.340076    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb7435c6-b83a-479b-85aa-37264666c7a4-model-cache\") pod \"sub-test-model-3682ae-kserve-55b8499769-w8j84\" (UID: \"bb7435c6-b83a-479b-85aa-37264666c7a4\") " pod="ai-tenant-e2e-shared-a-3682ae/sub-test-model-3682ae-kserve-55b8499769-w8j84"
Jul 28 22:54:38.340174 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:54:38.340098    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb7435c6-b83a-479b-85aa-37264666c7a4-tls-certs\") pod \"sub-test-model-3682ae-kserve-55b8499769-w8j84\" (UID: \"bb7435c6-b83a-479b-85aa-37264666c7a4\") " pod="ai-tenant-e2e-shared-a-3682ae/sub-test-model-3682ae-kserve-55b8499769-w8j84"
Jul 28 22:54:38.441136 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:54:38.441101    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb7435c6-b83a-479b-85aa-37264666c7a4-tmp-dir\") pod \"sub-test-model-3682ae-kserve-55b8499769-w8j84\" (UID: \"bb7435c6-b83a-479b-85aa-37264666c7a4\") " pod="ai-tenant-e2e-shared-a-3682ae/sub-test-model-3682ae-kserve-55b8499769-w8j84"
Jul 28 22:54:38.441340 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:54:38.441147    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbqc2\" (UniqueName: \"kubernetes.io/projected/bb7435c6-b83a-479b-85aa-37264666c7a4-kube-api-access-mbqc2\") pod \"sub-test-model-3682ae-kserve-55b8499769-w8j84\" (UID: \"bb7435c6-b83a-479b-85aa-37264666c7a4\") " pod="ai-tenant-e2e-shared-a-3682ae/sub-test-model-3682ae-kserve-55b8499769-w8j84"
Jul 28 22:54:38.441340 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:54:38.441200    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb7435c6-b83a-479b-85aa-37264666c7a4-model-cache\") pod \"sub-test-model-3682ae-kserve-55b8499769-w8j84\" (UID: \"bb7435c6-b83a-479b-85aa-37264666c7a4\") " pod="ai-tenant-e2e-shared-a-3682ae/sub-test-model-3682ae-kserve-55b8499769-w8j84"
Jul 28 22:54:38.441340 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:54:38.441222    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb7435c6-b83a-479b-85aa-37264666c7a4-tls-certs\") pod \"sub-test-model-3682ae-kserve-55b8499769-w8j84\" (UID: \"bb7435c6-b83a-479b-85aa-37264666c7a4\") " pod="ai-tenant-e2e-shared-a-3682ae/sub-test-model-3682ae-kserve-55b8499769-w8j84"
Jul 28 22:54:38.441340 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:54:38.441282    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb7435c6-b83a-479b-85aa-37264666c7a4-home\") pod \"sub-test-model-3682ae-kserve-55b8499769-w8j84\" (UID: \"bb7435c6-b83a-479b-85aa-37264666c7a4\") " pod="ai-tenant-e2e-shared-a-3682ae/sub-test-model-3682ae-kserve-55b8499769-w8j84"
Jul 28 22:54:38.441595 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:54:38.441347    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb7435c6-b83a-479b-85aa-37264666c7a4-dshm\") pod \"sub-test-model-3682ae-kserve-55b8499769-w8j84\" (UID: \"bb7435c6-b83a-479b-85aa-37264666c7a4\") " pod="ai-tenant-e2e-shared-a-3682ae/sub-test-model-3682ae-kserve-55b8499769-w8j84"
Jul 28 22:54:38.441635 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:54:38.441597    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb7435c6-b83a-479b-85aa-37264666c7a4-tmp-dir\") pod \"sub-test-model-3682ae-kserve-55b8499769-w8j84\" (UID: \"bb7435c6-b83a-479b-85aa-37264666c7a4\") " pod="ai-tenant-e2e-shared-a-3682ae/sub-test-model-3682ae-kserve-55b8499769-w8j84"
Jul 28 22:54:38.441752 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:54:38.441728    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb7435c6-b83a-479b-85aa-37264666c7a4-home\") pod \"sub-test-model-3682ae-kserve-55b8499769-w8j84\" (UID: \"bb7435c6-b83a-479b-85aa-37264666c7a4\") " pod="ai-tenant-e2e-shared-a-3682ae/sub-test-model-3682ae-kserve-55b8499769-w8j84"
Jul 28 22:54:38.441991 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:54:38.441967    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb7435c6-b83a-479b-85aa-37264666c7a4-model-cache\") pod \"sub-test-model-3682ae-kserve-55b8499769-w8j84\" (UID: \"bb7435c6-b83a-479b-85aa-37264666c7a4\") " pod="ai-tenant-e2e-shared-a-3682ae/sub-test-model-3682ae-kserve-55b8499769-w8j84"
Jul 28 22:54:38.443694 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:54:38.443666    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb7435c6-b83a-479b-85aa-37264666c7a4-dshm\") pod \"sub-test-model-3682ae-kserve-55b8499769-w8j84\" (UID: \"bb7435c6-b83a-479b-85aa-37264666c7a4\") " pod="ai-tenant-e2e-shared-a-3682ae/sub-test-model-3682ae-kserve-55b8499769-w8j84"
Jul 28 22:54:38.443900 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:54:38.443879    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb7435c6-b83a-479b-85aa-37264666c7a4-tls-certs\") pod \"sub-test-model-3682ae-kserve-55b8499769-w8j84\" (UID: \"bb7435c6-b83a-479b-85aa-37264666c7a4\") " pod="ai-tenant-e2e-shared-a-3682ae/sub-test-model-3682ae-kserve-55b8499769-w8j84"
Jul 28 22:54:38.449300 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:54:38.449279    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbqc2\" (UniqueName: \"kubernetes.io/projected/bb7435c6-b83a-479b-85aa-37264666c7a4-kube-api-access-mbqc2\") pod \"sub-test-model-3682ae-kserve-55b8499769-w8j84\" (UID: \"bb7435c6-b83a-479b-85aa-37264666c7a4\") " pod="ai-tenant-e2e-shared-a-3682ae/sub-test-model-3682ae-kserve-55b8499769-w8j84"
Jul 28 22:54:38.585363 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:54:38.585253    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3682ae/sub-test-model-3682ae-kserve-55b8499769-w8j84"
Jul 28 22:54:38.722649 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:54:38.722624    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-3682ae/sub-test-model-3682ae-kserve-55b8499769-w8j84"]
Jul 28 22:54:38.724820 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:54:38.724794    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb7435c6_b83a_479b_85aa_37264666c7a4.slice/crio-b4ef0c842a71fd6e461eade055be4b4e2282039ebef0c6798f84b640cc51f549 WatchSource:0}: Error finding container b4ef0c842a71fd6e461eade055be4b4e2282039ebef0c6798f84b640cc51f549: Status 404 returned error can't find the container with id b4ef0c842a71fd6e461eade055be4b4e2282039ebef0c6798f84b640cc51f549
Jul 28 22:54:39.544997 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:54:39.544959    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3682ae/sub-test-model-3682ae-kserve-55b8499769-w8j84" event={"ID":"bb7435c6-b83a-479b-85aa-37264666c7a4","Type":"ContainerStarted","Data":"d5f1be67c90d4da64f79f23a8ec2d973bb8bee383da32a84501dd63fd471c708"}
Jul 28 22:54:39.544997 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:54:39.545003    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3682ae/sub-test-model-3682ae-kserve-55b8499769-w8j84" event={"ID":"bb7435c6-b83a-479b-85aa-37264666c7a4","Type":"ContainerStarted","Data":"b4ef0c842a71fd6e461eade055be4b4e2282039ebef0c6798f84b640cc51f549"}
Jul 28 22:54:39.561456 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:54:39.561404    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-3682ae/sub-test-model-3682ae-kserve-55b8499769-w8j84" podStartSLOduration=1.561388466 podStartE2EDuration="1.561388466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:54:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:54:39.560940063 +0000 UTC m=+2976.941077711" watchObservedRunningTime="2026-07-28 22:54:39.561388466 +0000 UTC m=+2976.941526110"
Jul 28 22:54:48.586212 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:54:48.586177    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-3682ae/sub-test-model-3682ae-kserve-55b8499769-w8j84"
Jul 28 22:54:48.586731 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:54:48.586229    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-3682ae/sub-test-model-3682ae-kserve-55b8499769-w8j84"
Jul 28 22:54:48.599551 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:54:48.599527    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-3682ae/sub-test-model-3682ae-kserve-55b8499769-w8j84"
Jul 28 22:54:49.607295 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:54:49.607268    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-3682ae/sub-test-model-3682ae-kserve-55b8499769-w8j84"
Jul 28 22:55:03.243490 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:55:03.243463    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mcj8p_7bab32b0-13c6-4055-916e-a7454bafb5b6/ovn-acl-logging/0.log"
Jul 28 22:55:03.247994 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:55:03.247961    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mcj8p_7bab32b0-13c6-4055-916e-a7454bafb5b6/ovn-acl-logging/0.log"
Jul 28 22:56:16.790172 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:56:16.790087    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-3682ae/sub-test-model-3682ae-kserve-55b8499769-w8j84"]
Jul 28 22:56:31.809694 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:56:31.809649    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-3682ae/sub-test-model-3682ae-kserve-55b8499769-w8j84" podUID="bb7435c6-b83a-479b-85aa-37264666c7a4" containerName="main" containerID="cri-o://d5f1be67c90d4da64f79f23a8ec2d973bb8bee383da32a84501dd63fd471c708" gracePeriod=45
Jul 28 22:56:32.070461 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:56:32.070402    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3682ae/sub-test-model-3682ae-kserve-55b8499769-w8j84"
Jul 28 22:56:32.092060 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:56:32.092027    2563 generic.go:356] "Generic (PLEG): container finished" podID="bb7435c6-b83a-479b-85aa-37264666c7a4" containerID="d5f1be67c90d4da64f79f23a8ec2d973bb8bee383da32a84501dd63fd471c708" exitCode=0
Jul 28 22:56:32.092230 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:56:32.092074    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3682ae/sub-test-model-3682ae-kserve-55b8499769-w8j84"
Jul 28 22:56:32.092230 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:56:32.092109    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3682ae/sub-test-model-3682ae-kserve-55b8499769-w8j84" event={"ID":"bb7435c6-b83a-479b-85aa-37264666c7a4","Type":"ContainerDied","Data":"d5f1be67c90d4da64f79f23a8ec2d973bb8bee383da32a84501dd63fd471c708"}
Jul 28 22:56:32.092230 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:56:32.092145    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3682ae/sub-test-model-3682ae-kserve-55b8499769-w8j84" event={"ID":"bb7435c6-b83a-479b-85aa-37264666c7a4","Type":"ContainerDied","Data":"b4ef0c842a71fd6e461eade055be4b4e2282039ebef0c6798f84b640cc51f549"}
Jul 28 22:56:32.092230 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:56:32.092162    2563 scope.go:117] "RemoveContainer" containerID="d5f1be67c90d4da64f79f23a8ec2d973bb8bee383da32a84501dd63fd471c708"
Jul 28 22:56:32.101435 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:56:32.101409    2563 scope.go:117] "RemoveContainer" containerID="d5f1be67c90d4da64f79f23a8ec2d973bb8bee383da32a84501dd63fd471c708"
Jul 28 22:56:32.101715 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:56:32.101694    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5f1be67c90d4da64f79f23a8ec2d973bb8bee383da32a84501dd63fd471c708\": container with ID starting with d5f1be67c90d4da64f79f23a8ec2d973bb8bee383da32a84501dd63fd471c708 not found: ID does not exist" containerID="d5f1be67c90d4da64f79f23a8ec2d973bb8bee383da32a84501dd63fd471c708"
Jul 28 22:56:32.101798 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:56:32.101723    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5f1be67c90d4da64f79f23a8ec2d973bb8bee383da32a84501dd63fd471c708"} err="failed to get container status \"d5f1be67c90d4da64f79f23a8ec2d973bb8bee383da32a84501dd63fd471c708\": rpc error: code = NotFound desc = could not find container \"d5f1be67c90d4da64f79f23a8ec2d973bb8bee383da32a84501dd63fd471c708\": container with ID starting with d5f1be67c90d4da64f79f23a8ec2d973bb8bee383da32a84501dd63fd471c708 not found: ID does not exist"
Jul 28 22:56:32.151300 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:56:32.151277    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb7435c6-b83a-479b-85aa-37264666c7a4-dshm\") pod \"bb7435c6-b83a-479b-85aa-37264666c7a4\" (UID: \"bb7435c6-b83a-479b-85aa-37264666c7a4\") "
Jul 28 22:56:32.151447 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:56:32.151314    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb7435c6-b83a-479b-85aa-37264666c7a4-home\") pod \"bb7435c6-b83a-479b-85aa-37264666c7a4\" (UID: \"bb7435c6-b83a-479b-85aa-37264666c7a4\") "
Jul 28 22:56:32.151447 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:56:32.151391    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb7435c6-b83a-479b-85aa-37264666c7a4-tmp-dir\") pod \"bb7435c6-b83a-479b-85aa-37264666c7a4\" (UID: \"bb7435c6-b83a-479b-85aa-37264666c7a4\") "
Jul 28 22:56:32.151447 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:56:32.151411    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbqc2\" (UniqueName: \"kubernetes.io/projected/bb7435c6-b83a-479b-85aa-37264666c7a4-kube-api-access-mbqc2\") pod \"bb7435c6-b83a-479b-85aa-37264666c7a4\" (UID: \"bb7435c6-b83a-479b-85aa-37264666c7a4\") "
Jul 28 22:56:32.151447 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:56:32.151442    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb7435c6-b83a-479b-85aa-37264666c7a4-model-cache\") pod \"bb7435c6-b83a-479b-85aa-37264666c7a4\" (UID: \"bb7435c6-b83a-479b-85aa-37264666c7a4\") "
Jul 28 22:56:32.151665 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:56:32.151539    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb7435c6-b83a-479b-85aa-37264666c7a4-tls-certs\") pod \"bb7435c6-b83a-479b-85aa-37264666c7a4\" (UID: \"bb7435c6-b83a-479b-85aa-37264666c7a4\") "
Jul 28 22:56:32.151665 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:56:32.151623    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7435c6-b83a-479b-85aa-37264666c7a4-home" (OuterVolumeSpecName: "home") pod "bb7435c6-b83a-479b-85aa-37264666c7a4" (UID: "bb7435c6-b83a-479b-85aa-37264666c7a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:56:32.151780 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:56:32.151689    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7435c6-b83a-479b-85aa-37264666c7a4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bb7435c6-b83a-479b-85aa-37264666c7a4" (UID: "bb7435c6-b83a-479b-85aa-37264666c7a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:56:32.151895 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:56:32.151850    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7435c6-b83a-479b-85aa-37264666c7a4-model-cache" (OuterVolumeSpecName: "model-cache") pod "bb7435c6-b83a-479b-85aa-37264666c7a4" (UID: "bb7435c6-b83a-479b-85aa-37264666c7a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:56:32.151895 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:56:32.151880    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb7435c6-b83a-479b-85aa-37264666c7a4-home\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:56:32.152076 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:56:32.151902    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb7435c6-b83a-479b-85aa-37264666c7a4-tmp-dir\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:56:32.153595 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:56:32.153550    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7435c6-b83a-479b-85aa-37264666c7a4-dshm" (OuterVolumeSpecName: "dshm") pod "bb7435c6-b83a-479b-85aa-37264666c7a4" (UID: "bb7435c6-b83a-479b-85aa-37264666c7a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 22:56:32.153695 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:56:32.153618    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb7435c6-b83a-479b-85aa-37264666c7a4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bb7435c6-b83a-479b-85aa-37264666c7a4" (UID: "bb7435c6-b83a-479b-85aa-37264666c7a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 22:56:32.153695 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:56:32.153643    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb7435c6-b83a-479b-85aa-37264666c7a4-kube-api-access-mbqc2" (OuterVolumeSpecName: "kube-api-access-mbqc2") pod "bb7435c6-b83a-479b-85aa-37264666c7a4" (UID: "bb7435c6-b83a-479b-85aa-37264666c7a4"). InnerVolumeSpecName "kube-api-access-mbqc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 22:56:32.253273 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:56:32.253235    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbqc2\" (UniqueName: \"kubernetes.io/projected/bb7435c6-b83a-479b-85aa-37264666c7a4-kube-api-access-mbqc2\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:56:32.253273 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:56:32.253264    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb7435c6-b83a-479b-85aa-37264666c7a4-model-cache\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:56:32.253273 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:56:32.253280    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb7435c6-b83a-479b-85aa-37264666c7a4-tls-certs\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:56:32.253546 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:56:32.253300    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb7435c6-b83a-479b-85aa-37264666c7a4-dshm\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 22:56:32.416232 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:56:32.416165    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-3682ae/sub-test-model-3682ae-kserve-55b8499769-w8j84"]
Jul 28 22:56:32.418118 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:56:32.418095    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-3682ae/sub-test-model-3682ae-kserve-55b8499769-w8j84"]
Jul 28 22:56:33.130406 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:56:33.130371    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb7435c6-b83a-479b-85aa-37264666c7a4" path="/var/lib/kubelet/pods/bb7435c6-b83a-479b-85aa-37264666c7a4/volumes"
Jul 28 22:58:36.805835 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:36.805802    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebba94-74696989d9-p2scc"]
Jul 28 22:58:36.806624 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:36.806244    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb7435c6-b83a-479b-85aa-37264666c7a4" containerName="main"
Jul 28 22:58:36.806624 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:36.806257    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb7435c6-b83a-479b-85aa-37264666c7a4" containerName="main"
Jul 28 22:58:36.806624 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:36.806320    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb7435c6-b83a-479b-85aa-37264666c7a4" containerName="main"
Jul 28 22:58:36.809604 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:36.809583    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebba94-74696989d9-p2scc"
Jul 28 22:58:36.812045 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:36.812021    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-ebba94-serving-cert\""
Jul 28 22:58:36.819824 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:36.819803    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebba94-74696989d9-p2scc"]
Jul 28 22:58:36.955731 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:36.955698    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h49kj\" (UniqueName: \"kubernetes.io/projected/da61fa5c-8cd3-4869-9abd-c53c74ce93d5-kube-api-access-h49kj\") pod \"maas-api-e2e-ipp-a-ebba94-74696989d9-p2scc\" (UID: \"da61fa5c-8cd3-4869-9abd-c53c74ce93d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebba94-74696989d9-p2scc"
Jul 28 22:58:36.955899 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:36.955805    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da61fa5c-8cd3-4869-9abd-c53c74ce93d5-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ebba94-74696989d9-p2scc\" (UID: \"da61fa5c-8cd3-4869-9abd-c53c74ce93d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebba94-74696989d9-p2scc"
Jul 28 22:58:36.955899 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:36.955838    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da61fa5c-8cd3-4869-9abd-c53c74ce93d5-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ebba94-74696989d9-p2scc\" (UID: \"da61fa5c-8cd3-4869-9abd-c53c74ce93d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebba94-74696989d9-p2scc"
Jul 28 22:58:37.056820 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:37.056730    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da61fa5c-8cd3-4869-9abd-c53c74ce93d5-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ebba94-74696989d9-p2scc\" (UID: \"da61fa5c-8cd3-4869-9abd-c53c74ce93d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebba94-74696989d9-p2scc"
Jul 28 22:58:37.056820 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:37.056771    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da61fa5c-8cd3-4869-9abd-c53c74ce93d5-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ebba94-74696989d9-p2scc\" (UID: \"da61fa5c-8cd3-4869-9abd-c53c74ce93d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebba94-74696989d9-p2scc"
Jul 28 22:58:37.057019 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:37.056829    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h49kj\" (UniqueName: \"kubernetes.io/projected/da61fa5c-8cd3-4869-9abd-c53c74ce93d5-kube-api-access-h49kj\") pod \"maas-api-e2e-ipp-a-ebba94-74696989d9-p2scc\" (UID: \"da61fa5c-8cd3-4869-9abd-c53c74ce93d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebba94-74696989d9-p2scc"
Jul 28 22:58:37.057019 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:58:37.056888    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebba94-serving-cert: secret "maas-api-e2e-ipp-a-ebba94-serving-cert" not found
Jul 28 22:58:37.057019 ip-10-0-128-37 kubenswrapper[2563]: E0728 22:58:37.056955    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da61fa5c-8cd3-4869-9abd-c53c74ce93d5-maas-api-tls podName:da61fa5c-8cd3-4869-9abd-c53c74ce93d5 nodeName:}" failed. No retries permitted until 2026-07-28 22:58:37.556939554 +0000 UTC m=+3214.937077176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/da61fa5c-8cd3-4869-9abd-c53c74ce93d5-maas-api-tls") pod "maas-api-e2e-ipp-a-ebba94-74696989d9-p2scc" (UID: "da61fa5c-8cd3-4869-9abd-c53c74ce93d5") : secret "maas-api-e2e-ipp-a-ebba94-serving-cert" not found
Jul 28 22:58:37.057524 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:37.057506    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da61fa5c-8cd3-4869-9abd-c53c74ce93d5-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ebba94-74696989d9-p2scc\" (UID: \"da61fa5c-8cd3-4869-9abd-c53c74ce93d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebba94-74696989d9-p2scc"
Jul 28 22:58:37.079367 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:37.079325    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h49kj\" (UniqueName: \"kubernetes.io/projected/da61fa5c-8cd3-4869-9abd-c53c74ce93d5-kube-api-access-h49kj\") pod \"maas-api-e2e-ipp-a-ebba94-74696989d9-p2scc\" (UID: \"da61fa5c-8cd3-4869-9abd-c53c74ce93d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebba94-74696989d9-p2scc"
Jul 28 22:58:37.561635 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:37.561599    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da61fa5c-8cd3-4869-9abd-c53c74ce93d5-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ebba94-74696989d9-p2scc\" (UID: \"da61fa5c-8cd3-4869-9abd-c53c74ce93d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebba94-74696989d9-p2scc"
Jul 28 22:58:37.564064 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:37.564035    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da61fa5c-8cd3-4869-9abd-c53c74ce93d5-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ebba94-74696989d9-p2scc\" (UID: \"da61fa5c-8cd3-4869-9abd-c53c74ce93d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebba94-74696989d9-p2scc"
Jul 28 22:58:37.653515 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:37.653481    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ebba94-c545f456d-jb5kj"]
Jul 28 22:58:37.657079 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:37.657057    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebba94-c545f456d-jb5kj"
Jul 28 22:58:37.662317 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:37.662289    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ebba94\""
Jul 28 22:58:37.674658 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:37.674634    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ebba94-c545f456d-jb5kj"]
Jul 28 22:58:37.721668 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:37.721635    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebba94-74696989d9-p2scc"
Jul 28 22:58:37.763878 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:37.763837    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d61132a7-a786-4441-8c5c-c31fcea80679-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ebba94-c545f456d-jb5kj\" (UID: \"d61132a7-a786-4441-8c5c-c31fcea80679\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebba94-c545f456d-jb5kj"
Jul 28 22:58:37.764039 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:37.763903    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tns6m\" (UniqueName: \"kubernetes.io/projected/d61132a7-a786-4441-8c5c-c31fcea80679-kube-api-access-tns6m\") pod \"payload-pre-processing-e2e-ipp-a-ebba94-c545f456d-jb5kj\" (UID: \"d61132a7-a786-4441-8c5c-c31fcea80679\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebba94-c545f456d-jb5kj"
Jul 28 22:58:37.864587 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:37.864546    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tns6m\" (UniqueName: \"kubernetes.io/projected/d61132a7-a786-4441-8c5c-c31fcea80679-kube-api-access-tns6m\") pod \"payload-pre-processing-e2e-ipp-a-ebba94-c545f456d-jb5kj\" (UID: \"d61132a7-a786-4441-8c5c-c31fcea80679\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebba94-c545f456d-jb5kj"
Jul 28 22:58:37.864964 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:37.864701    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d61132a7-a786-4441-8c5c-c31fcea80679-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ebba94-c545f456d-jb5kj\" (UID: \"d61132a7-a786-4441-8c5c-c31fcea80679\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebba94-c545f456d-jb5kj"
Jul 28 22:58:37.865618 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:37.865593    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d61132a7-a786-4441-8c5c-c31fcea80679-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ebba94-c545f456d-jb5kj\" (UID: \"d61132a7-a786-4441-8c5c-c31fcea80679\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebba94-c545f456d-jb5kj"
Jul 28 22:58:37.871261 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:37.871240    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebba94-74696989d9-p2scc"]
Jul 28 22:58:37.873639 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:58:37.873615    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda61fa5c_8cd3_4869_9abd_c53c74ce93d5.slice/crio-001d7e66dbff1aa1cc82baef458c0783502426b436de97742332ef926568b63e WatchSource:0}: Error finding container 001d7e66dbff1aa1cc82baef458c0783502426b436de97742332ef926568b63e: Status 404 returned error can't find the container with id 001d7e66dbff1aa1cc82baef458c0783502426b436de97742332ef926568b63e
Jul 28 22:58:37.875263 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:37.875235    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tns6m\" (UniqueName: \"kubernetes.io/projected/d61132a7-a786-4441-8c5c-c31fcea80679-kube-api-access-tns6m\") pod \"payload-pre-processing-e2e-ipp-a-ebba94-c545f456d-jb5kj\" (UID: \"d61132a7-a786-4441-8c5c-c31fcea80679\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebba94-c545f456d-jb5kj"
Jul 28 22:58:37.875405 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:37.875389    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 22:58:37.967925 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:37.967886    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebba94-c545f456d-jb5kj"
Jul 28 22:58:38.314630 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:38.314606    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ebba94-c545f456d-jb5kj"]
Jul 28 22:58:38.317516 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:58:38.317491    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd61132a7_a786_4441_8c5c_c31fcea80679.slice/crio-274dc069da246ecb2e4470de09980ccdae10ac32c6135935716b5b01cfec3cc1 WatchSource:0}: Error finding container 274dc069da246ecb2e4470de09980ccdae10ac32c6135935716b5b01cfec3cc1: Status 404 returned error can't find the container with id 274dc069da246ecb2e4470de09980ccdae10ac32c6135935716b5b01cfec3cc1
Jul 28 22:58:38.708700 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:38.708658    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebba94-74696989d9-p2scc" event={"ID":"da61fa5c-8cd3-4869-9abd-c53c74ce93d5","Type":"ContainerStarted","Data":"66db9350d55903808565b4b1521a474a540db585f25a2286d31101abe10e7933"}
Jul 28 22:58:38.708914 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:38.708730    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebba94-74696989d9-p2scc" event={"ID":"da61fa5c-8cd3-4869-9abd-c53c74ce93d5","Type":"ContainerStarted","Data":"001d7e66dbff1aa1cc82baef458c0783502426b436de97742332ef926568b63e"}
Jul 28 22:58:38.708994 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:38.708933    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebba94-74696989d9-p2scc"
Jul 28 22:58:38.710405 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:38.710380    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebba94-c545f456d-jb5kj" event={"ID":"d61132a7-a786-4441-8c5c-c31fcea80679","Type":"ContainerStarted","Data":"467c54653cd4dc407dac8dc7ac2be5819851c5e6e70def37ac159bfe50854e5a"}
Jul 28 22:58:38.710516 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:38.710424    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebba94-c545f456d-jb5kj" event={"ID":"d61132a7-a786-4441-8c5c-c31fcea80679","Type":"ContainerStarted","Data":"274dc069da246ecb2e4470de09980ccdae10ac32c6135935716b5b01cfec3cc1"}
Jul 28 22:58:38.710516 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:38.710445    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebba94-c545f456d-jb5kj"
Jul 28 22:58:38.727311 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:38.727264    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebba94-74696989d9-p2scc" podStartSLOduration=2.326633067 podStartE2EDuration="2.727250568s" totalImagesPullingTime="400.617501ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:58:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:58:38.725123719 +0000 UTC m=+3216.105261365" watchObservedRunningTime="2026-07-28 22:58:38.727250568 +0000 UTC m=+3216.107388236"
Jul 28 22:58:38.742083 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:38.742033    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebba94-c545f456d-jb5kj" podStartSLOduration=1.742017471 podStartE2EDuration="1.742017471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:58:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:58:38.740187551 +0000 UTC m=+3216.120325210" watchObservedRunningTime="2026-07-28 22:58:38.742017471 +0000 UTC m=+3216.122155117"
Jul 28 22:58:44.721232 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:44.721200    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebba94-74696989d9-p2scc"
Jul 28 22:58:49.717882 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:58:49.717849    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebba94-c545f456d-jb5kj"
Jul 28 22:59:06.876764 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:59:06.876683    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-908e19-66d8f994dd-8css9"]
Jul 28 22:59:06.880464 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:59:06.880440    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908e19-66d8f994dd-8css9"
Jul 28 22:59:06.883705 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:59:06.883674    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-908e19\""
Jul 28 22:59:06.885935 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:59:06.885899    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-908e19-dockercfg-h8mc7\""
Jul 28 22:59:06.894280 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:59:06.894252    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-908e19-66d8f994dd-8css9"]
Jul 28 22:59:06.902317 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:59:06.902295    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-908e19-7ccfc5bfc6-kxdp2"]
Jul 28 22:59:06.905771 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:59:06.905753    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-908e19-7ccfc5bfc6-kxdp2"
Jul 28 22:59:06.912058 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:59:06.912032    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5sqf\" (UniqueName: \"kubernetes.io/projected/b3670d0d-3532-4f09-9804-e65ad2ea648c-kube-api-access-g5sqf\") pod \"payload-pre-processing-e2e-ipp-b-908e19-66d8f994dd-8css9\" (UID: \"b3670d0d-3532-4f09-9804-e65ad2ea648c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908e19-66d8f994dd-8css9"
Jul 28 22:59:06.912298 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:59:06.912185    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3670d0d-3532-4f09-9804-e65ad2ea648c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-908e19-66d8f994dd-8css9\" (UID: \"b3670d0d-3532-4f09-9804-e65ad2ea648c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908e19-66d8f994dd-8css9"
Jul 28 22:59:06.919472 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:59:06.919442    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-908e19-7ccfc5bfc6-kxdp2"]
Jul 28 22:59:07.012682 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:59:07.012642    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5sqf\" (UniqueName: \"kubernetes.io/projected/b3670d0d-3532-4f09-9804-e65ad2ea648c-kube-api-access-g5sqf\") pod \"payload-pre-processing-e2e-ipp-b-908e19-66d8f994dd-8css9\" (UID: \"b3670d0d-3532-4f09-9804-e65ad2ea648c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908e19-66d8f994dd-8css9"
Jul 28 22:59:07.012682 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:59:07.012687    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16f9d31b-a579-4187-8d9d-6a0bbf2919f7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-908e19-7ccfc5bfc6-kxdp2\" (UID: \"16f9d31b-a579-4187-8d9d-6a0bbf2919f7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-908e19-7ccfc5bfc6-kxdp2"
Jul 28 22:59:07.012905 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:59:07.012726    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krgxd\" (UniqueName: \"kubernetes.io/projected/16f9d31b-a579-4187-8d9d-6a0bbf2919f7-kube-api-access-krgxd\") pod \"payload-processing-e2e-ipp-b-908e19-7ccfc5bfc6-kxdp2\" (UID: \"16f9d31b-a579-4187-8d9d-6a0bbf2919f7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-908e19-7ccfc5bfc6-kxdp2"
Jul 28 22:59:07.012905 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:59:07.012785    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3670d0d-3532-4f09-9804-e65ad2ea648c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-908e19-66d8f994dd-8css9\" (UID: \"b3670d0d-3532-4f09-9804-e65ad2ea648c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908e19-66d8f994dd-8css9"
Jul 28 22:59:07.013501 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:59:07.013483    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3670d0d-3532-4f09-9804-e65ad2ea648c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-908e19-66d8f994dd-8css9\" (UID: \"b3670d0d-3532-4f09-9804-e65ad2ea648c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908e19-66d8f994dd-8css9"
Jul 28 22:59:07.022876 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:59:07.022849    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5sqf\" (UniqueName: \"kubernetes.io/projected/b3670d0d-3532-4f09-9804-e65ad2ea648c-kube-api-access-g5sqf\") pod \"payload-pre-processing-e2e-ipp-b-908e19-66d8f994dd-8css9\" (UID: \"b3670d0d-3532-4f09-9804-e65ad2ea648c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908e19-66d8f994dd-8css9"
Jul 28 22:59:07.113940 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:59:07.113903    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16f9d31b-a579-4187-8d9d-6a0bbf2919f7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-908e19-7ccfc5bfc6-kxdp2\" (UID: \"16f9d31b-a579-4187-8d9d-6a0bbf2919f7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-908e19-7ccfc5bfc6-kxdp2"
Jul 28 22:59:07.114087 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:59:07.113966    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krgxd\" (UniqueName: \"kubernetes.io/projected/16f9d31b-a579-4187-8d9d-6a0bbf2919f7-kube-api-access-krgxd\") pod \"payload-processing-e2e-ipp-b-908e19-7ccfc5bfc6-kxdp2\" (UID: \"16f9d31b-a579-4187-8d9d-6a0bbf2919f7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-908e19-7ccfc5bfc6-kxdp2"
Jul 28 22:59:07.114664 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:59:07.114645    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16f9d31b-a579-4187-8d9d-6a0bbf2919f7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-908e19-7ccfc5bfc6-kxdp2\" (UID: \"16f9d31b-a579-4187-8d9d-6a0bbf2919f7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-908e19-7ccfc5bfc6-kxdp2"
Jul 28 22:59:07.128508 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:59:07.128454    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krgxd\" (UniqueName: \"kubernetes.io/projected/16f9d31b-a579-4187-8d9d-6a0bbf2919f7-kube-api-access-krgxd\") pod \"payload-processing-e2e-ipp-b-908e19-7ccfc5bfc6-kxdp2\" (UID: \"16f9d31b-a579-4187-8d9d-6a0bbf2919f7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-908e19-7ccfc5bfc6-kxdp2"
Jul 28 22:59:07.190985 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:59:07.190949    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908e19-66d8f994dd-8css9"
Jul 28 22:59:07.216247 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:59:07.216222    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-908e19-7ccfc5bfc6-kxdp2"
Jul 28 22:59:07.547218 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:59:07.547187    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-908e19-66d8f994dd-8css9"]
Jul 28 22:59:07.549114 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:59:07.549087    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3670d0d_3532_4f09_9804_e65ad2ea648c.slice/crio-69186ba0792cea1a9285e5d58aa490dbfbf3654c3137c60db7bc365ae160e465 WatchSource:0}: Error finding container 69186ba0792cea1a9285e5d58aa490dbfbf3654c3137c60db7bc365ae160e465: Status 404 returned error can't find the container with id 69186ba0792cea1a9285e5d58aa490dbfbf3654c3137c60db7bc365ae160e465
Jul 28 22:59:07.580804 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:59:07.580774    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-908e19-7ccfc5bfc6-kxdp2"]
Jul 28 22:59:07.587551 ip-10-0-128-37 kubenswrapper[2563]: W0728 22:59:07.587524    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16f9d31b_a579_4187_8d9d_6a0bbf2919f7.slice/crio-9ebb931848af30d545c1ad243f0f89dc18335561bab70b52d2daedbd185c883b WatchSource:0}: Error finding container 9ebb931848af30d545c1ad243f0f89dc18335561bab70b52d2daedbd185c883b: Status 404 returned error can't find the container with id 9ebb931848af30d545c1ad243f0f89dc18335561bab70b52d2daedbd185c883b
Jul 28 22:59:07.861099 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:59:07.860994    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-908e19-7ccfc5bfc6-kxdp2" event={"ID":"16f9d31b-a579-4187-8d9d-6a0bbf2919f7","Type":"ContainerStarted","Data":"08877e158ff3b2f0bd8a94f659f13cb8b9232ca002c653b1b7173361fbaa1c81"}
Jul 28 22:59:07.861099 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:59:07.861060    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-908e19-7ccfc5bfc6-kxdp2" event={"ID":"16f9d31b-a579-4187-8d9d-6a0bbf2919f7","Type":"ContainerStarted","Data":"9ebb931848af30d545c1ad243f0f89dc18335561bab70b52d2daedbd185c883b"}
Jul 28 22:59:07.861099 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:59:07.861090    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-908e19-7ccfc5bfc6-kxdp2"
Jul 28 22:59:07.862692 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:59:07.862664    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908e19-66d8f994dd-8css9" event={"ID":"b3670d0d-3532-4f09-9804-e65ad2ea648c","Type":"ContainerStarted","Data":"7cfd77df2a384d08b5ecbc7de0990533d947eeb92fa505e92253dc94b21916cf"}
Jul 28 22:59:07.862796 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:59:07.862712    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908e19-66d8f994dd-8css9"
Jul 28 22:59:07.862796 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:59:07.862733    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908e19-66d8f994dd-8css9" event={"ID":"b3670d0d-3532-4f09-9804-e65ad2ea648c","Type":"ContainerStarted","Data":"69186ba0792cea1a9285e5d58aa490dbfbf3654c3137c60db7bc365ae160e465"}
Jul 28 22:59:07.879481 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:59:07.879440    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-908e19-7ccfc5bfc6-kxdp2" podStartSLOduration=1.879427464 podStartE2EDuration="1.879427464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:59:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:59:07.877444562 +0000 UTC m=+3245.257582207" watchObservedRunningTime="2026-07-28 22:59:07.879427464 +0000 UTC m=+3245.259565108"
Jul 28 22:59:07.894537 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:59:07.894496    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908e19-66d8f994dd-8css9" podStartSLOduration=1.894481192 podStartE2EDuration="1.894481192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:59:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:59:07.892631981 +0000 UTC m=+3245.272769639" watchObservedRunningTime="2026-07-28 22:59:07.894481192 +0000 UTC m=+3245.274618835"
Jul 28 22:59:18.870240 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:59:18.870193    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908e19-66d8f994dd-8css9" podUID="b3670d0d-3532-4f09-9804-e65ad2ea648c" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.80:9004: connect: connection refused"
Jul 28 22:59:18.870928 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:59:18.870486    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-908e19-7ccfc5bfc6-kxdp2"
Jul 28 22:59:28.869599 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:59:28.869560    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908e19-66d8f994dd-8css9"
Jul 28 22:59:37.853247 ip-10-0-128-37 kubenswrapper[2563]: I0728 22:59:37.853218    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-908e19-7ccfc5bfc6-kxdp2_16f9d31b-a579-4187-8d9d-6a0bbf2919f7/payload-processing/0.log"
Jul 28 23:00:03.273933 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:03.273901    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mcj8p_7bab32b0-13c6-4055-916e-a7454bafb5b6/ovn-acl-logging/0.log"
Jul 28 23:00:03.280230 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:03.280207    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mcj8p_7bab32b0-13c6-4055-916e-a7454bafb5b6/ovn-acl-logging/0.log"
Jul 28 23:00:30.281318 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:30.281238    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-908e19-7ccfc5bfc6-kxdp2_16f9d31b-a579-4187-8d9d-6a0bbf2919f7/payload-processing/0.log"
Jul 28 23:00:46.142819 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:46.142783    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-76d3be-7fd6d686fd-fgcq4"]
Jul 28 23:00:46.146557 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:46.146534    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-76d3be-7fd6d686fd-fgcq4"
Jul 28 23:00:46.154397 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:46.150857    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-76d3be-serving-cert\""
Jul 28 23:00:46.159001 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:46.158645    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-76d3be-7fd6d686fd-fgcq4"]
Jul 28 23:00:46.224265 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:46.224232    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-76d3be-7fd6d686fd-fgcq4\" (UID: \"1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-76d3be-7fd6d686fd-fgcq4"
Jul 28 23:00:46.224513 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:46.224378    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-76d3be-7fd6d686fd-fgcq4\" (UID: \"1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-76d3be-7fd6d686fd-fgcq4"
Jul 28 23:00:46.224513 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:46.224418    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vbnr\" (UniqueName: \"kubernetes.io/projected/1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a-kube-api-access-9vbnr\") pod \"maas-api-e2e-ipp-cleanup-76d3be-7fd6d686fd-fgcq4\" (UID: \"1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-76d3be-7fd6d686fd-fgcq4"
Jul 28 23:00:46.325755 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:46.325705    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-76d3be-7fd6d686fd-fgcq4\" (UID: \"1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-76d3be-7fd6d686fd-fgcq4"
Jul 28 23:00:46.325755 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:46.325764    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vbnr\" (UniqueName: \"kubernetes.io/projected/1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a-kube-api-access-9vbnr\") pod \"maas-api-e2e-ipp-cleanup-76d3be-7fd6d686fd-fgcq4\" (UID: \"1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-76d3be-7fd6d686fd-fgcq4"
Jul 28 23:00:46.325977 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:46.325820    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-76d3be-7fd6d686fd-fgcq4\" (UID: \"1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-76d3be-7fd6d686fd-fgcq4"
Jul 28 23:00:46.325977 ip-10-0-128-37 kubenswrapper[2563]: E0728 23:00:46.325948    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-76d3be-serving-cert: secret "maas-api-e2e-ipp-cleanup-76d3be-serving-cert" not found
Jul 28 23:00:46.326049 ip-10-0-128-37 kubenswrapper[2563]: E0728 23:00:46.326007    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a-maas-api-tls podName:1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a nodeName:}" failed. No retries permitted until 2026-07-28 23:00:46.825988494 +0000 UTC m=+3344.206126120 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-76d3be-7fd6d686fd-fgcq4" (UID: "1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a") : secret "maas-api-e2e-ipp-cleanup-76d3be-serving-cert" not found
Jul 28 23:00:46.326375 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:46.326334    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-76d3be-7fd6d686fd-fgcq4\" (UID: \"1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-76d3be-7fd6d686fd-fgcq4"
Jul 28 23:00:46.335226 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:46.335192    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vbnr\" (UniqueName: \"kubernetes.io/projected/1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a-kube-api-access-9vbnr\") pod \"maas-api-e2e-ipp-cleanup-76d3be-7fd6d686fd-fgcq4\" (UID: \"1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-76d3be-7fd6d686fd-fgcq4"
Jul 28 23:00:46.831179 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:46.831141    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-76d3be-7fd6d686fd-fgcq4\" (UID: \"1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-76d3be-7fd6d686fd-fgcq4"
Jul 28 23:00:46.833741 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:46.833713    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-76d3be-7fd6d686fd-fgcq4\" (UID: \"1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-76d3be-7fd6d686fd-fgcq4"
Jul 28 23:00:46.877113 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:46.877075    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-76d3be-7d649c99d4-wkldc"]
Jul 28 23:00:46.880900 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:46.880881    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-76d3be-7d649c99d4-wkldc"
Jul 28 23:00:46.883330 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:46.883308    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-76d3be\""
Jul 28 23:00:46.883557 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:46.883532    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-76d3be-dockercfg-jlng9\""
Jul 28 23:00:46.890822 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:46.890798    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-76d3be-7d649c99d4-wkldc"]
Jul 28 23:00:46.932280 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:46.932249    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aede49d6-041a-48c4-bd47-8364a383a641-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-76d3be-7d649c99d4-wkldc\" (UID: \"aede49d6-041a-48c4-bd47-8364a383a641\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-76d3be-7d649c99d4-wkldc"
Jul 28 23:00:46.932450 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:46.932310    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpvvn\" (UniqueName: \"kubernetes.io/projected/aede49d6-041a-48c4-bd47-8364a383a641-kube-api-access-zpvvn\") pod \"payload-pre-processing-e2e-ipp-cleanup-76d3be-7d649c99d4-wkldc\" (UID: \"aede49d6-041a-48c4-bd47-8364a383a641\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-76d3be-7d649c99d4-wkldc"
Jul 28 23:00:47.033094 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:47.033050    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpvvn\" (UniqueName: \"kubernetes.io/projected/aede49d6-041a-48c4-bd47-8364a383a641-kube-api-access-zpvvn\") pod \"payload-pre-processing-e2e-ipp-cleanup-76d3be-7d649c99d4-wkldc\" (UID: \"aede49d6-041a-48c4-bd47-8364a383a641\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-76d3be-7d649c99d4-wkldc"
Jul 28 23:00:47.033289 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:47.033167    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aede49d6-041a-48c4-bd47-8364a383a641-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-76d3be-7d649c99d4-wkldc\" (UID: \"aede49d6-041a-48c4-bd47-8364a383a641\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-76d3be-7d649c99d4-wkldc"
Jul 28 23:00:47.033913 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:47.033885    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aede49d6-041a-48c4-bd47-8364a383a641-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-76d3be-7d649c99d4-wkldc\" (UID: \"aede49d6-041a-48c4-bd47-8364a383a641\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-76d3be-7d649c99d4-wkldc"
Jul 28 23:00:47.041692 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:47.041659    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpvvn\" (UniqueName: \"kubernetes.io/projected/aede49d6-041a-48c4-bd47-8364a383a641-kube-api-access-zpvvn\") pod \"payload-pre-processing-e2e-ipp-cleanup-76d3be-7d649c99d4-wkldc\" (UID: \"aede49d6-041a-48c4-bd47-8364a383a641\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-76d3be-7d649c99d4-wkldc"
Jul 28 23:00:47.065079 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:47.065054    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-76d3be-7fd6d686fd-fgcq4"
Jul 28 23:00:47.194198 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:47.194167    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-76d3be-7d649c99d4-wkldc"
Jul 28 23:00:47.209996 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:47.209956    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-76d3be-7fd6d686fd-fgcq4"]
Jul 28 23:00:47.210227 ip-10-0-128-37 kubenswrapper[2563]: W0728 23:00:47.210197    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ffedae5_fde3_4a8d_90e2_4fb81da7aa2a.slice/crio-2d5f99f5cd3fb0d14c9a8b021db2b8a047de40c6d5c66e5e6805267170423f02 WatchSource:0}: Error finding container 2d5f99f5cd3fb0d14c9a8b021db2b8a047de40c6d5c66e5e6805267170423f02: Status 404 returned error can't find the container with id 2d5f99f5cd3fb0d14c9a8b021db2b8a047de40c6d5c66e5e6805267170423f02
Jul 28 23:00:47.358193 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:47.358157    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-76d3be-7fd6d686fd-fgcq4" event={"ID":"1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a","Type":"ContainerStarted","Data":"2d5f99f5cd3fb0d14c9a8b021db2b8a047de40c6d5c66e5e6805267170423f02"}
Jul 28 23:00:47.537623 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:47.537372    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-76d3be-7d649c99d4-wkldc"]
Jul 28 23:00:47.539666 ip-10-0-128-37 kubenswrapper[2563]: W0728 23:00:47.539637    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaede49d6_041a_48c4_bd47_8364a383a641.slice/crio-feeebf7c9d4a96f0d7909d29e2a13a4eb2d3f5c625153e8a75d20458474691a9 WatchSource:0}: Error finding container feeebf7c9d4a96f0d7909d29e2a13a4eb2d3f5c625153e8a75d20458474691a9: Status 404 returned error can't find the container with id feeebf7c9d4a96f0d7909d29e2a13a4eb2d3f5c625153e8a75d20458474691a9
Jul 28 23:00:48.364780 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:48.364695    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-76d3be-7fd6d686fd-fgcq4" event={"ID":"1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a","Type":"ContainerStarted","Data":"43a0f8d5aa7d654f0ee4f7316f799b88cc7c2bedc35f7b149d9db5001b7f5730"}
Jul 28 23:00:48.364780 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:48.364771    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-76d3be-7fd6d686fd-fgcq4"
Jul 28 23:00:48.366413 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:48.366378    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-76d3be-7d649c99d4-wkldc" event={"ID":"aede49d6-041a-48c4-bd47-8364a383a641","Type":"ContainerStarted","Data":"141f12bbbd2a944ed169697e3f1e0dbc04be85f838020beef425a780a6c38156"}
Jul 28 23:00:48.366549 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:48.366427    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-76d3be-7d649c99d4-wkldc" event={"ID":"aede49d6-041a-48c4-bd47-8364a383a641","Type":"ContainerStarted","Data":"feeebf7c9d4a96f0d7909d29e2a13a4eb2d3f5c625153e8a75d20458474691a9"}
Jul 28 23:00:48.366549 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:48.366447    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-76d3be-7d649c99d4-wkldc"
Jul 28 23:00:48.383782 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:48.383742    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-76d3be-7fd6d686fd-fgcq4" podStartSLOduration=1.5262835030000002 podStartE2EDuration="2.383728963s" totalImagesPullingTime="857.44546ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:00:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:00:48.381541717 +0000 UTC m=+3345.761679355" watchObservedRunningTime="2026-07-28 23:00:48.383728963 +0000 UTC m=+3345.763866607"
Jul 28 23:00:48.400426 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:48.400379    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-76d3be-7d649c99d4-wkldc" podStartSLOduration=2.400361842 podStartE2EDuration="2.400361842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:00:48.398474888 +0000 UTC m=+3345.778612558" watchObservedRunningTime="2026-07-28 23:00:48.400361842 +0000 UTC m=+3345.780499479"
Jul 28 23:00:54.377484 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:54.377454    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-76d3be-7fd6d686fd-fgcq4"
Jul 28 23:00:59.374123 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:00:59.374091    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-76d3be-7d649c99d4-wkldc"
Jul 28 23:01:08.368776 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:08.368737    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-76d3be-7fd6d686fd-fgcq4"]
Jul 28 23:01:08.369284 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:08.368984    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-76d3be-7fd6d686fd-fgcq4" podUID="1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a" containerName="maas-api" containerID="cri-o://43a0f8d5aa7d654f0ee4f7316f799b88cc7c2bedc35f7b149d9db5001b7f5730" gracePeriod=30
Jul 28 23:01:08.430801 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:08.430764    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-76d3be-7d649c99d4-wkldc"]
Jul 28 23:01:08.431048 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:08.431013    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-76d3be-7d649c99d4-wkldc" podUID="aede49d6-041a-48c4-bd47-8364a383a641" containerName="payload-pre-processing" containerID="cri-o://141f12bbbd2a944ed169697e3f1e0dbc04be85f838020beef425a780a6c38156" gracePeriod=30
Jul 28 23:01:08.691259 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:08.691223    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-76d3be-7fd6d686fd-fgcq4"
Jul 28 23:01:08.729301 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:08.729276    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-76d3be-7d649c99d4-wkldc"
Jul 28 23:01:08.740720 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:08.740691    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a-openshift-service-ca\") pod \"1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a\" (UID: \"1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a\") "
Jul 28 23:01:08.740874 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:08.740732    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a-maas-api-tls\") pod \"1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a\" (UID: \"1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a\") "
Jul 28 23:01:08.740874 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:08.740757    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aede49d6-041a-48c4-bd47-8364a383a641-plugins-config-volume\") pod \"aede49d6-041a-48c4-bd47-8364a383a641\" (UID: \"aede49d6-041a-48c4-bd47-8364a383a641\") "
Jul 28 23:01:08.740874 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:08.740834    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpvvn\" (UniqueName: \"kubernetes.io/projected/aede49d6-041a-48c4-bd47-8364a383a641-kube-api-access-zpvvn\") pod \"aede49d6-041a-48c4-bd47-8364a383a641\" (UID: \"aede49d6-041a-48c4-bd47-8364a383a641\") "
Jul 28 23:01:08.740874 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:08.740853    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vbnr\" (UniqueName: \"kubernetes.io/projected/1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a-kube-api-access-9vbnr\") pod \"1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a\" (UID: \"1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a\") "
Jul 28 23:01:08.741137 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:08.741098    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a" (UID: "1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:01:08.741200 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:08.741157    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aede49d6-041a-48c4-bd47-8364a383a641-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "aede49d6-041a-48c4-bd47-8364a383a641" (UID: "aede49d6-041a-48c4-bd47-8364a383a641"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:01:08.743066 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:08.743040    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a-kube-api-access-9vbnr" (OuterVolumeSpecName: "kube-api-access-9vbnr") pod "1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a" (UID: "1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a"). InnerVolumeSpecName "kube-api-access-9vbnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 23:01:08.743175 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:08.743092    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aede49d6-041a-48c4-bd47-8364a383a641-kube-api-access-zpvvn" (OuterVolumeSpecName: "kube-api-access-zpvvn") pod "aede49d6-041a-48c4-bd47-8364a383a641" (UID: "aede49d6-041a-48c4-bd47-8364a383a641"). InnerVolumeSpecName "kube-api-access-zpvvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 23:01:08.743233 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:08.743224    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a" (UID: "1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 23:01:08.842081 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:08.842037    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpvvn\" (UniqueName: \"kubernetes.io/projected/aede49d6-041a-48c4-bd47-8364a383a641-kube-api-access-zpvvn\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 23:01:08.842081 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:08.842071    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vbnr\" (UniqueName: \"kubernetes.io/projected/1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a-kube-api-access-9vbnr\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 23:01:08.842081 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:08.842081    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a-openshift-service-ca\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 23:01:08.842081 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:08.842091    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a-maas-api-tls\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 23:01:08.842400 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:08.842100    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aede49d6-041a-48c4-bd47-8364a383a641-plugins-config-volume\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 23:01:09.477758 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:09.477717    2563 generic.go:356] "Generic (PLEG): container finished" podID="1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a" containerID="43a0f8d5aa7d654f0ee4f7316f799b88cc7c2bedc35f7b149d9db5001b7f5730" exitCode=0
Jul 28 23:01:09.478213 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:09.477769    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-76d3be-7fd6d686fd-fgcq4"
Jul 28 23:01:09.478213 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:09.477800    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-76d3be-7fd6d686fd-fgcq4" event={"ID":"1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a","Type":"ContainerDied","Data":"43a0f8d5aa7d654f0ee4f7316f799b88cc7c2bedc35f7b149d9db5001b7f5730"}
Jul 28 23:01:09.478213 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:09.477867    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-76d3be-7fd6d686fd-fgcq4" event={"ID":"1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a","Type":"ContainerDied","Data":"2d5f99f5cd3fb0d14c9a8b021db2b8a047de40c6d5c66e5e6805267170423f02"}
Jul 28 23:01:09.478213 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:09.477902    2563 scope.go:117] "RemoveContainer" containerID="43a0f8d5aa7d654f0ee4f7316f799b88cc7c2bedc35f7b149d9db5001b7f5730"
Jul 28 23:01:09.479259 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:09.479238    2563 generic.go:356] "Generic (PLEG): container finished" podID="aede49d6-041a-48c4-bd47-8364a383a641" containerID="141f12bbbd2a944ed169697e3f1e0dbc04be85f838020beef425a780a6c38156" exitCode=0
Jul 28 23:01:09.479337 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:09.479290    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-76d3be-7d649c99d4-wkldc" event={"ID":"aede49d6-041a-48c4-bd47-8364a383a641","Type":"ContainerDied","Data":"141f12bbbd2a944ed169697e3f1e0dbc04be85f838020beef425a780a6c38156"}
Jul 28 23:01:09.479337 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:09.479300    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-76d3be-7d649c99d4-wkldc"
Jul 28 23:01:09.479337 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:09.479332    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-76d3be-7d649c99d4-wkldc" event={"ID":"aede49d6-041a-48c4-bd47-8364a383a641","Type":"ContainerDied","Data":"feeebf7c9d4a96f0d7909d29e2a13a4eb2d3f5c625153e8a75d20458474691a9"}
Jul 28 23:01:09.489400 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:09.489368    2563 scope.go:117] "RemoveContainer" containerID="43a0f8d5aa7d654f0ee4f7316f799b88cc7c2bedc35f7b149d9db5001b7f5730"
Jul 28 23:01:09.489677 ip-10-0-128-37 kubenswrapper[2563]: E0728 23:01:09.489657    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43a0f8d5aa7d654f0ee4f7316f799b88cc7c2bedc35f7b149d9db5001b7f5730\": container with ID starting with 43a0f8d5aa7d654f0ee4f7316f799b88cc7c2bedc35f7b149d9db5001b7f5730 not found: ID does not exist" containerID="43a0f8d5aa7d654f0ee4f7316f799b88cc7c2bedc35f7b149d9db5001b7f5730"
Jul 28 23:01:09.489769 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:09.489690    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43a0f8d5aa7d654f0ee4f7316f799b88cc7c2bedc35f7b149d9db5001b7f5730"} err="failed to get container status \"43a0f8d5aa7d654f0ee4f7316f799b88cc7c2bedc35f7b149d9db5001b7f5730\": rpc error: code = NotFound desc = could not find container \"43a0f8d5aa7d654f0ee4f7316f799b88cc7c2bedc35f7b149d9db5001b7f5730\": container with ID starting with 43a0f8d5aa7d654f0ee4f7316f799b88cc7c2bedc35f7b149d9db5001b7f5730 not found: ID does not exist"
Jul 28 23:01:09.489769 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:09.489713    2563 scope.go:117] "RemoveContainer" containerID="141f12bbbd2a944ed169697e3f1e0dbc04be85f838020beef425a780a6c38156"
Jul 28 23:01:09.497629 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:09.497609    2563 scope.go:117] "RemoveContainer" containerID="141f12bbbd2a944ed169697e3f1e0dbc04be85f838020beef425a780a6c38156"
Jul 28 23:01:09.497879 ip-10-0-128-37 kubenswrapper[2563]: E0728 23:01:09.497860    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"141f12bbbd2a944ed169697e3f1e0dbc04be85f838020beef425a780a6c38156\": container with ID starting with 141f12bbbd2a944ed169697e3f1e0dbc04be85f838020beef425a780a6c38156 not found: ID does not exist" containerID="141f12bbbd2a944ed169697e3f1e0dbc04be85f838020beef425a780a6c38156"
Jul 28 23:01:09.497928 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:09.497886    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"141f12bbbd2a944ed169697e3f1e0dbc04be85f838020beef425a780a6c38156"} err="failed to get container status \"141f12bbbd2a944ed169697e3f1e0dbc04be85f838020beef425a780a6c38156\": rpc error: code = NotFound desc = could not find container \"141f12bbbd2a944ed169697e3f1e0dbc04be85f838020beef425a780a6c38156\": container with ID starting with 141f12bbbd2a944ed169697e3f1e0dbc04be85f838020beef425a780a6c38156 not found: ID does not exist"
Jul 28 23:01:09.504073 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:09.504053    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-76d3be-7fd6d686fd-fgcq4"]
Jul 28 23:01:09.507685 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:09.507662    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-76d3be-7fd6d686fd-fgcq4"]
Jul 28 23:01:09.517550 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:09.517532    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-76d3be-7d649c99d4-wkldc"]
Jul 28 23:01:09.521290 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:09.521269    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-76d3be-7d649c99d4-wkldc"]
Jul 28 23:01:11.131060 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:11.131024    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a" path="/var/lib/kubelet/pods/1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a/volumes"
Jul 28 23:01:11.131520 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:11.131502    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aede49d6-041a-48c4-bd47-8364a383a641" path="/var/lib/kubelet/pods/aede49d6-041a-48c4-bd47-8364a383a641/volumes"
Jul 28 23:01:33.951512 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:33.951471    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebba94-74696989d9-p2scc"]
Jul 28 23:01:33.952097 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:33.951731    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebba94-74696989d9-p2scc" podUID="da61fa5c-8cd3-4869-9abd-c53c74ce93d5" containerName="maas-api" containerID="cri-o://66db9350d55903808565b4b1521a474a540db585f25a2286d31101abe10e7933" gracePeriod=30
Jul 28 23:01:34.007876 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:34.007842    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ebba94-c545f456d-jb5kj"]
Jul 28 23:01:34.008160 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:34.008097    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebba94-c545f456d-jb5kj" podUID="d61132a7-a786-4441-8c5c-c31fcea80679" containerName="payload-pre-processing" containerID="cri-o://467c54653cd4dc407dac8dc7ac2be5819851c5e6e70def37ac159bfe50854e5a" gracePeriod=30
Jul 28 23:01:34.274930 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:34.274902    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebba94-74696989d9-p2scc"
Jul 28 23:01:34.310286 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:34.310223    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebba94-c545f456d-jb5kj"
Jul 28 23:01:34.374406 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:34.374372    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da61fa5c-8cd3-4869-9abd-c53c74ce93d5-openshift-service-ca\") pod \"da61fa5c-8cd3-4869-9abd-c53c74ce93d5\" (UID: \"da61fa5c-8cd3-4869-9abd-c53c74ce93d5\") "
Jul 28 23:01:34.374598 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:34.374427    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h49kj\" (UniqueName: \"kubernetes.io/projected/da61fa5c-8cd3-4869-9abd-c53c74ce93d5-kube-api-access-h49kj\") pod \"da61fa5c-8cd3-4869-9abd-c53c74ce93d5\" (UID: \"da61fa5c-8cd3-4869-9abd-c53c74ce93d5\") "
Jul 28 23:01:34.374598 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:34.374456    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da61fa5c-8cd3-4869-9abd-c53c74ce93d5-maas-api-tls\") pod \"da61fa5c-8cd3-4869-9abd-c53c74ce93d5\" (UID: \"da61fa5c-8cd3-4869-9abd-c53c74ce93d5\") "
Jul 28 23:01:34.374598 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:34.374519    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tns6m\" (UniqueName: \"kubernetes.io/projected/d61132a7-a786-4441-8c5c-c31fcea80679-kube-api-access-tns6m\") pod \"d61132a7-a786-4441-8c5c-c31fcea80679\" (UID: \"d61132a7-a786-4441-8c5c-c31fcea80679\") "
Jul 28 23:01:34.374598 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:34.374593    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d61132a7-a786-4441-8c5c-c31fcea80679-plugins-config-volume\") pod \"d61132a7-a786-4441-8c5c-c31fcea80679\" (UID: \"d61132a7-a786-4441-8c5c-c31fcea80679\") "
Jul 28 23:01:34.374829 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:34.374807    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da61fa5c-8cd3-4869-9abd-c53c74ce93d5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "da61fa5c-8cd3-4869-9abd-c53c74ce93d5" (UID: "da61fa5c-8cd3-4869-9abd-c53c74ce93d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:01:34.375000 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:34.374978    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da61fa5c-8cd3-4869-9abd-c53c74ce93d5-openshift-service-ca\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 23:01:34.375091 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:34.375034    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d61132a7-a786-4441-8c5c-c31fcea80679-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d61132a7-a786-4441-8c5c-c31fcea80679" (UID: "d61132a7-a786-4441-8c5c-c31fcea80679"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:01:34.376625 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:34.376601    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da61fa5c-8cd3-4869-9abd-c53c74ce93d5-kube-api-access-h49kj" (OuterVolumeSpecName: "kube-api-access-h49kj") pod "da61fa5c-8cd3-4869-9abd-c53c74ce93d5" (UID: "da61fa5c-8cd3-4869-9abd-c53c74ce93d5"). InnerVolumeSpecName "kube-api-access-h49kj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 23:01:34.376695 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:34.376620    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d61132a7-a786-4441-8c5c-c31fcea80679-kube-api-access-tns6m" (OuterVolumeSpecName: "kube-api-access-tns6m") pod "d61132a7-a786-4441-8c5c-c31fcea80679" (UID: "d61132a7-a786-4441-8c5c-c31fcea80679"). InnerVolumeSpecName "kube-api-access-tns6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 23:01:34.376695 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:34.376606    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da61fa5c-8cd3-4869-9abd-c53c74ce93d5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "da61fa5c-8cd3-4869-9abd-c53c74ce93d5" (UID: "da61fa5c-8cd3-4869-9abd-c53c74ce93d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 23:01:34.475563 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:34.475523    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tns6m\" (UniqueName: \"kubernetes.io/projected/d61132a7-a786-4441-8c5c-c31fcea80679-kube-api-access-tns6m\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 23:01:34.475563 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:34.475549    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d61132a7-a786-4441-8c5c-c31fcea80679-plugins-config-volume\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 23:01:34.475563 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:34.475559    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h49kj\" (UniqueName: \"kubernetes.io/projected/da61fa5c-8cd3-4869-9abd-c53c74ce93d5-kube-api-access-h49kj\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 23:01:34.475563 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:34.475570    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da61fa5c-8cd3-4869-9abd-c53c74ce93d5-maas-api-tls\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 23:01:34.607746 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:34.607640    2563 generic.go:356] "Generic (PLEG): container finished" podID="da61fa5c-8cd3-4869-9abd-c53c74ce93d5" containerID="66db9350d55903808565b4b1521a474a540db585f25a2286d31101abe10e7933" exitCode=0
Jul 28 23:01:34.607746 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:34.607705    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebba94-74696989d9-p2scc"
Jul 28 23:01:34.607989 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:34.607740    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebba94-74696989d9-p2scc" event={"ID":"da61fa5c-8cd3-4869-9abd-c53c74ce93d5","Type":"ContainerDied","Data":"66db9350d55903808565b4b1521a474a540db585f25a2286d31101abe10e7933"}
Jul 28 23:01:34.607989 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:34.607803    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebba94-74696989d9-p2scc" event={"ID":"da61fa5c-8cd3-4869-9abd-c53c74ce93d5","Type":"ContainerDied","Data":"001d7e66dbff1aa1cc82baef458c0783502426b436de97742332ef926568b63e"}
Jul 28 23:01:34.607989 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:34.607834    2563 scope.go:117] "RemoveContainer" containerID="66db9350d55903808565b4b1521a474a540db585f25a2286d31101abe10e7933"
Jul 28 23:01:34.609074 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:34.609047    2563 generic.go:356] "Generic (PLEG): container finished" podID="d61132a7-a786-4441-8c5c-c31fcea80679" containerID="467c54653cd4dc407dac8dc7ac2be5819851c5e6e70def37ac159bfe50854e5a" exitCode=0
Jul 28 23:01:34.609190 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:34.609085    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebba94-c545f456d-jb5kj" event={"ID":"d61132a7-a786-4441-8c5c-c31fcea80679","Type":"ContainerDied","Data":"467c54653cd4dc407dac8dc7ac2be5819851c5e6e70def37ac159bfe50854e5a"}
Jul 28 23:01:34.609190 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:34.609104    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebba94-c545f456d-jb5kj"
Jul 28 23:01:34.609190 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:34.609134    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebba94-c545f456d-jb5kj" event={"ID":"d61132a7-a786-4441-8c5c-c31fcea80679","Type":"ContainerDied","Data":"274dc069da246ecb2e4470de09980ccdae10ac32c6135935716b5b01cfec3cc1"}
Jul 28 23:01:34.616919 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:34.616900    2563 scope.go:117] "RemoveContainer" containerID="66db9350d55903808565b4b1521a474a540db585f25a2286d31101abe10e7933"
Jul 28 23:01:34.617218 ip-10-0-128-37 kubenswrapper[2563]: E0728 23:01:34.617193    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66db9350d55903808565b4b1521a474a540db585f25a2286d31101abe10e7933\": container with ID starting with 66db9350d55903808565b4b1521a474a540db585f25a2286d31101abe10e7933 not found: ID does not exist" containerID="66db9350d55903808565b4b1521a474a540db585f25a2286d31101abe10e7933"
Jul 28 23:01:34.617286 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:34.617230    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66db9350d55903808565b4b1521a474a540db585f25a2286d31101abe10e7933"} err="failed to get container status \"66db9350d55903808565b4b1521a474a540db585f25a2286d31101abe10e7933\": rpc error: code = NotFound desc = could not find container \"66db9350d55903808565b4b1521a474a540db585f25a2286d31101abe10e7933\": container with ID starting with 66db9350d55903808565b4b1521a474a540db585f25a2286d31101abe10e7933 not found: ID does not exist"
Jul 28 23:01:34.617286 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:34.617254    2563 scope.go:117] "RemoveContainer" containerID="467c54653cd4dc407dac8dc7ac2be5819851c5e6e70def37ac159bfe50854e5a"
Jul 28 23:01:34.625172 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:34.625154    2563 scope.go:117] "RemoveContainer" containerID="467c54653cd4dc407dac8dc7ac2be5819851c5e6e70def37ac159bfe50854e5a"
Jul 28 23:01:34.625432 ip-10-0-128-37 kubenswrapper[2563]: E0728 23:01:34.625415    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"467c54653cd4dc407dac8dc7ac2be5819851c5e6e70def37ac159bfe50854e5a\": container with ID starting with 467c54653cd4dc407dac8dc7ac2be5819851c5e6e70def37ac159bfe50854e5a not found: ID does not exist" containerID="467c54653cd4dc407dac8dc7ac2be5819851c5e6e70def37ac159bfe50854e5a"
Jul 28 23:01:34.625481 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:34.625438    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"467c54653cd4dc407dac8dc7ac2be5819851c5e6e70def37ac159bfe50854e5a"} err="failed to get container status \"467c54653cd4dc407dac8dc7ac2be5819851c5e6e70def37ac159bfe50854e5a\": rpc error: code = NotFound desc = could not find container \"467c54653cd4dc407dac8dc7ac2be5819851c5e6e70def37ac159bfe50854e5a\": container with ID starting with 467c54653cd4dc407dac8dc7ac2be5819851c5e6e70def37ac159bfe50854e5a not found: ID does not exist"
Jul 28 23:01:34.631947 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:34.631924    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ebba94-c545f456d-jb5kj"]
Jul 28 23:01:34.635894 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:34.635874    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ebba94-c545f456d-jb5kj"]
Jul 28 23:01:34.645534 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:34.645511    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebba94-74696989d9-p2scc"]
Jul 28 23:01:34.648826 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:34.648805    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ebba94-74696989d9-p2scc"]
Jul 28 23:01:35.131104 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:35.131067    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d61132a7-a786-4441-8c5c-c31fcea80679" path="/var/lib/kubelet/pods/d61132a7-a786-4441-8c5c-c31fcea80679/volumes"
Jul 28 23:01:35.131503 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:01:35.131482    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da61fa5c-8cd3-4869-9abd-c53c74ce93d5" path="/var/lib/kubelet/pods/da61fa5c-8cd3-4869-9abd-c53c74ce93d5/volumes"
Jul 28 23:02:30.041413 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:30.041379    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-908e19-7ccfc5bfc6-kxdp2"]
Jul 28 23:02:30.041944 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:30.041607    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-908e19-7ccfc5bfc6-kxdp2" podUID="16f9d31b-a579-4187-8d9d-6a0bbf2919f7" containerName="payload-processing" containerID="cri-o://08877e158ff3b2f0bd8a94f659f13cb8b9232ca002c653b1b7173361fbaa1c81" gracePeriod=30
Jul 28 23:02:30.047924 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:30.047897    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-908e19-66d8f994dd-8css9"]
Jul 28 23:02:30.048203 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:30.048177    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908e19-66d8f994dd-8css9" podUID="b3670d0d-3532-4f09-9804-e65ad2ea648c" containerName="payload-pre-processing" containerID="cri-o://7cfd77df2a384d08b5ecbc7de0990533d947eeb92fa505e92253dc94b21916cf" gracePeriod=30
Jul 28 23:02:30.350875 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:30.350830    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-908e19-7ccfc5bfc6-kxdp2"
Jul 28 23:02:30.355898 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:30.355534    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908e19-66d8f994dd-8css9"
Jul 28 23:02:30.496404 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:30.496342    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16f9d31b-a579-4187-8d9d-6a0bbf2919f7-plugins-config-volume\") pod \"16f9d31b-a579-4187-8d9d-6a0bbf2919f7\" (UID: \"16f9d31b-a579-4187-8d9d-6a0bbf2919f7\") "
Jul 28 23:02:30.496609 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:30.496421    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3670d0d-3532-4f09-9804-e65ad2ea648c-plugins-config-volume\") pod \"b3670d0d-3532-4f09-9804-e65ad2ea648c\" (UID: \"b3670d0d-3532-4f09-9804-e65ad2ea648c\") "
Jul 28 23:02:30.496609 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:30.496512    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5sqf\" (UniqueName: \"kubernetes.io/projected/b3670d0d-3532-4f09-9804-e65ad2ea648c-kube-api-access-g5sqf\") pod \"b3670d0d-3532-4f09-9804-e65ad2ea648c\" (UID: \"b3670d0d-3532-4f09-9804-e65ad2ea648c\") "
Jul 28 23:02:30.496609 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:30.496581    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krgxd\" (UniqueName: \"kubernetes.io/projected/16f9d31b-a579-4187-8d9d-6a0bbf2919f7-kube-api-access-krgxd\") pod \"16f9d31b-a579-4187-8d9d-6a0bbf2919f7\" (UID: \"16f9d31b-a579-4187-8d9d-6a0bbf2919f7\") "
Jul 28 23:02:30.496775 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:30.496747    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16f9d31b-a579-4187-8d9d-6a0bbf2919f7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "16f9d31b-a579-4187-8d9d-6a0bbf2919f7" (UID: "16f9d31b-a579-4187-8d9d-6a0bbf2919f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:02:30.496844 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:30.496818    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3670d0d-3532-4f09-9804-e65ad2ea648c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b3670d0d-3532-4f09-9804-e65ad2ea648c" (UID: "b3670d0d-3532-4f09-9804-e65ad2ea648c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:02:30.496968 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:30.496944    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16f9d31b-a579-4187-8d9d-6a0bbf2919f7-plugins-config-volume\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 23:02:30.496968 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:30.496964    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3670d0d-3532-4f09-9804-e65ad2ea648c-plugins-config-volume\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 23:02:30.498719 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:30.498696    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3670d0d-3532-4f09-9804-e65ad2ea648c-kube-api-access-g5sqf" (OuterVolumeSpecName: "kube-api-access-g5sqf") pod "b3670d0d-3532-4f09-9804-e65ad2ea648c" (UID: "b3670d0d-3532-4f09-9804-e65ad2ea648c"). InnerVolumeSpecName "kube-api-access-g5sqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 23:02:30.498845 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:30.498724    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16f9d31b-a579-4187-8d9d-6a0bbf2919f7-kube-api-access-krgxd" (OuterVolumeSpecName: "kube-api-access-krgxd") pod "16f9d31b-a579-4187-8d9d-6a0bbf2919f7" (UID: "16f9d31b-a579-4187-8d9d-6a0bbf2919f7"). InnerVolumeSpecName "kube-api-access-krgxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 23:02:30.597830 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:30.597742    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5sqf\" (UniqueName: \"kubernetes.io/projected/b3670d0d-3532-4f09-9804-e65ad2ea648c-kube-api-access-g5sqf\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 23:02:30.597830 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:30.597773    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krgxd\" (UniqueName: \"kubernetes.io/projected/16f9d31b-a579-4187-8d9d-6a0bbf2919f7-kube-api-access-krgxd\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 23:02:30.913767 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:30.913665    2563 generic.go:356] "Generic (PLEG): container finished" podID="16f9d31b-a579-4187-8d9d-6a0bbf2919f7" containerID="08877e158ff3b2f0bd8a94f659f13cb8b9232ca002c653b1b7173361fbaa1c81" exitCode=0
Jul 28 23:02:30.913767 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:30.913737    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-908e19-7ccfc5bfc6-kxdp2"
Jul 28 23:02:30.913767 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:30.913749    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-908e19-7ccfc5bfc6-kxdp2" event={"ID":"16f9d31b-a579-4187-8d9d-6a0bbf2919f7","Type":"ContainerDied","Data":"08877e158ff3b2f0bd8a94f659f13cb8b9232ca002c653b1b7173361fbaa1c81"}
Jul 28 23:02:30.914050 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:30.913860    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-908e19-7ccfc5bfc6-kxdp2" event={"ID":"16f9d31b-a579-4187-8d9d-6a0bbf2919f7","Type":"ContainerDied","Data":"9ebb931848af30d545c1ad243f0f89dc18335561bab70b52d2daedbd185c883b"}
Jul 28 23:02:30.914050 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:30.913897    2563 scope.go:117] "RemoveContainer" containerID="08877e158ff3b2f0bd8a94f659f13cb8b9232ca002c653b1b7173361fbaa1c81"
Jul 28 23:02:30.915192 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:30.915172    2563 generic.go:356] "Generic (PLEG): container finished" podID="b3670d0d-3532-4f09-9804-e65ad2ea648c" containerID="7cfd77df2a384d08b5ecbc7de0990533d947eeb92fa505e92253dc94b21916cf" exitCode=0
Jul 28 23:02:30.915275 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:30.915235    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908e19-66d8f994dd-8css9"
Jul 28 23:02:30.915275 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:30.915248    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908e19-66d8f994dd-8css9" event={"ID":"b3670d0d-3532-4f09-9804-e65ad2ea648c","Type":"ContainerDied","Data":"7cfd77df2a384d08b5ecbc7de0990533d947eeb92fa505e92253dc94b21916cf"}
Jul 28 23:02:30.915413 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:30.915280    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908e19-66d8f994dd-8css9" event={"ID":"b3670d0d-3532-4f09-9804-e65ad2ea648c","Type":"ContainerDied","Data":"69186ba0792cea1a9285e5d58aa490dbfbf3654c3137c60db7bc365ae160e465"}
Jul 28 23:02:30.923932 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:30.923914    2563 scope.go:117] "RemoveContainer" containerID="08877e158ff3b2f0bd8a94f659f13cb8b9232ca002c653b1b7173361fbaa1c81"
Jul 28 23:02:30.924206 ip-10-0-128-37 kubenswrapper[2563]: E0728 23:02:30.924186    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08877e158ff3b2f0bd8a94f659f13cb8b9232ca002c653b1b7173361fbaa1c81\": container with ID starting with 08877e158ff3b2f0bd8a94f659f13cb8b9232ca002c653b1b7173361fbaa1c81 not found: ID does not exist" containerID="08877e158ff3b2f0bd8a94f659f13cb8b9232ca002c653b1b7173361fbaa1c81"
Jul 28 23:02:30.924265 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:30.924214    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08877e158ff3b2f0bd8a94f659f13cb8b9232ca002c653b1b7173361fbaa1c81"} err="failed to get container status \"08877e158ff3b2f0bd8a94f659f13cb8b9232ca002c653b1b7173361fbaa1c81\": rpc error: code = NotFound desc = could not find container \"08877e158ff3b2f0bd8a94f659f13cb8b9232ca002c653b1b7173361fbaa1c81\": container with ID starting with 08877e158ff3b2f0bd8a94f659f13cb8b9232ca002c653b1b7173361fbaa1c81 not found: ID does not exist"
Jul 28 23:02:30.924265 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:30.924233    2563 scope.go:117] "RemoveContainer" containerID="7cfd77df2a384d08b5ecbc7de0990533d947eeb92fa505e92253dc94b21916cf"
Jul 28 23:02:30.932486 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:30.932469    2563 scope.go:117] "RemoveContainer" containerID="7cfd77df2a384d08b5ecbc7de0990533d947eeb92fa505e92253dc94b21916cf"
Jul 28 23:02:30.932792 ip-10-0-128-37 kubenswrapper[2563]: E0728 23:02:30.932773    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cfd77df2a384d08b5ecbc7de0990533d947eeb92fa505e92253dc94b21916cf\": container with ID starting with 7cfd77df2a384d08b5ecbc7de0990533d947eeb92fa505e92253dc94b21916cf not found: ID does not exist" containerID="7cfd77df2a384d08b5ecbc7de0990533d947eeb92fa505e92253dc94b21916cf"
Jul 28 23:02:30.932851 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:30.932800    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cfd77df2a384d08b5ecbc7de0990533d947eeb92fa505e92253dc94b21916cf"} err="failed to get container status \"7cfd77df2a384d08b5ecbc7de0990533d947eeb92fa505e92253dc94b21916cf\": rpc error: code = NotFound desc = could not find container \"7cfd77df2a384d08b5ecbc7de0990533d947eeb92fa505e92253dc94b21916cf\": container with ID starting with 7cfd77df2a384d08b5ecbc7de0990533d947eeb92fa505e92253dc94b21916cf not found: ID does not exist"
Jul 28 23:02:30.938252 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:30.938225    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-908e19-7ccfc5bfc6-kxdp2"]
Jul 28 23:02:30.942522 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:30.942498    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-908e19-7ccfc5bfc6-kxdp2"]
Jul 28 23:02:30.951914 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:30.951892    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-908e19-66d8f994dd-8css9"]
Jul 28 23:02:30.956548 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:30.956523    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-908e19-66d8f994dd-8css9"]
Jul 28 23:02:31.130202 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:31.130172    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16f9d31b-a579-4187-8d9d-6a0bbf2919f7" path="/var/lib/kubelet/pods/16f9d31b-a579-4187-8d9d-6a0bbf2919f7/volumes"
Jul 28 23:02:31.130605 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:31.130557    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3670d0d-3532-4f09-9804-e65ad2ea648c" path="/var/lib/kubelet/pods/b3670d0d-3532-4f09-9804-e65ad2ea648c/volumes"
Jul 28 23:02:44.306005 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.305968    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-3682ae-5f995b88d9-vclmw"]
Jul 28 23:02:44.306612 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.306189    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3682ae-5f995b88d9-vclmw" podUID="0a2a293b-ce61-4e53-8821-79c92242ae07" containerName="maas-api" containerID="cri-o://b73573f908364ba30d1db57bf2e07abcfb79d63a0a2648aba9df64643023eb75" gracePeriod=30
Jul 28 23:02:44.359578 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.359544    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-3682ae-68d4966958-dnpv4"]
Jul 28 23:02:44.359812 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.359787    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-3682ae-68d4966958-dnpv4" podUID="7480613f-9415-433c-b63c-72501451864d" containerName="payload-processing" containerID="cri-o://84b33b2e7c0da322da02d7fdcff6390ae5994b9d6c8de35ddee4053951d3070d" gracePeriod=30
Jul 28 23:02:44.368455 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.368420    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-3682ae-7985768976-vtr77"]
Jul 28 23:02:44.368668 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.368640    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-3682ae-7985768976-vtr77" podUID="acc7d7c1-cb50-4705-9676-3fedeacdda8c" containerName="payload-pre-processing" containerID="cri-o://6f4df81472038ccee17dbd75e1b258d10f5bf3ca26fe17a6ce93f1c1aa717a12" gracePeriod=30
Jul 28 23:02:44.654787 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.654756    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-3682ae-68d4966958-dnpv4"
Jul 28 23:02:44.670628 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.670606    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3682ae-5f995b88d9-vclmw"
Jul 28 23:02:44.681075 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.681051    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-3682ae-7985768976-vtr77"
Jul 28 23:02:44.727997 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.727965    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dt7kg\" (UniqueName: \"kubernetes.io/projected/0a2a293b-ce61-4e53-8821-79c92242ae07-kube-api-access-dt7kg\") pod \"0a2a293b-ce61-4e53-8821-79c92242ae07\" (UID: \"0a2a293b-ce61-4e53-8821-79c92242ae07\") "
Jul 28 23:02:44.728150 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.728021    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7480613f-9415-433c-b63c-72501451864d-plugins-config-volume\") pod \"7480613f-9415-433c-b63c-72501451864d\" (UID: \"7480613f-9415-433c-b63c-72501451864d\") "
Jul 28 23:02:44.728150 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.728047    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acc7d7c1-cb50-4705-9676-3fedeacdda8c-plugins-config-volume\") pod \"acc7d7c1-cb50-4705-9676-3fedeacdda8c\" (UID: \"acc7d7c1-cb50-4705-9676-3fedeacdda8c\") "
Jul 28 23:02:44.728293 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.728148    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shcnr\" (UniqueName: \"kubernetes.io/projected/7480613f-9415-433c-b63c-72501451864d-kube-api-access-shcnr\") pod \"7480613f-9415-433c-b63c-72501451864d\" (UID: \"7480613f-9415-433c-b63c-72501451864d\") "
Jul 28 23:02:44.728293 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.728192    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9lzl\" (UniqueName: \"kubernetes.io/projected/acc7d7c1-cb50-4705-9676-3fedeacdda8c-kube-api-access-j9lzl\") pod \"acc7d7c1-cb50-4705-9676-3fedeacdda8c\" (UID: \"acc7d7c1-cb50-4705-9676-3fedeacdda8c\") "
Jul 28 23:02:44.728293 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.728234    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a2a293b-ce61-4e53-8821-79c92242ae07-maas-api-tls\") pod \"0a2a293b-ce61-4e53-8821-79c92242ae07\" (UID: \"0a2a293b-ce61-4e53-8821-79c92242ae07\") "
Jul 28 23:02:44.728293 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.728261    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a2a293b-ce61-4e53-8821-79c92242ae07-openshift-service-ca\") pod \"0a2a293b-ce61-4e53-8821-79c92242ae07\" (UID: \"0a2a293b-ce61-4e53-8821-79c92242ae07\") "
Jul 28 23:02:44.728549 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.728417    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acc7d7c1-cb50-4705-9676-3fedeacdda8c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "acc7d7c1-cb50-4705-9676-3fedeacdda8c" (UID: "acc7d7c1-cb50-4705-9676-3fedeacdda8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:02:44.728549 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.728465    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7480613f-9415-433c-b63c-72501451864d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7480613f-9415-433c-b63c-72501451864d" (UID: "7480613f-9415-433c-b63c-72501451864d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:02:44.728772 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.728750    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acc7d7c1-cb50-4705-9676-3fedeacdda8c-plugins-config-volume\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 23:02:44.728835 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.728787    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7480613f-9415-433c-b63c-72501451864d-plugins-config-volume\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 23:02:44.728835 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.728755    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a2a293b-ce61-4e53-8821-79c92242ae07-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0a2a293b-ce61-4e53-8821-79c92242ae07" (UID: "0a2a293b-ce61-4e53-8821-79c92242ae07"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:02:44.730323 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.730290    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7480613f-9415-433c-b63c-72501451864d-kube-api-access-shcnr" (OuterVolumeSpecName: "kube-api-access-shcnr") pod "7480613f-9415-433c-b63c-72501451864d" (UID: "7480613f-9415-433c-b63c-72501451864d"). InnerVolumeSpecName "kube-api-access-shcnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 23:02:44.730323 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.730295    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acc7d7c1-cb50-4705-9676-3fedeacdda8c-kube-api-access-j9lzl" (OuterVolumeSpecName: "kube-api-access-j9lzl") pod "acc7d7c1-cb50-4705-9676-3fedeacdda8c" (UID: "acc7d7c1-cb50-4705-9676-3fedeacdda8c"). InnerVolumeSpecName "kube-api-access-j9lzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 23:02:44.730777 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.730744    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a2a293b-ce61-4e53-8821-79c92242ae07-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0a2a293b-ce61-4e53-8821-79c92242ae07" (UID: "0a2a293b-ce61-4e53-8821-79c92242ae07"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 23:02:44.730865 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.730751    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a2a293b-ce61-4e53-8821-79c92242ae07-kube-api-access-dt7kg" (OuterVolumeSpecName: "kube-api-access-dt7kg") pod "0a2a293b-ce61-4e53-8821-79c92242ae07" (UID: "0a2a293b-ce61-4e53-8821-79c92242ae07"). InnerVolumeSpecName "kube-api-access-dt7kg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 23:02:44.830197 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.830167    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9lzl\" (UniqueName: \"kubernetes.io/projected/acc7d7c1-cb50-4705-9676-3fedeacdda8c-kube-api-access-j9lzl\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 23:02:44.830197 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.830193    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a2a293b-ce61-4e53-8821-79c92242ae07-maas-api-tls\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 23:02:44.830197 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.830203    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a2a293b-ce61-4e53-8821-79c92242ae07-openshift-service-ca\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 23:02:44.830460 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.830211    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt7kg\" (UniqueName: \"kubernetes.io/projected/0a2a293b-ce61-4e53-8821-79c92242ae07-kube-api-access-dt7kg\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 23:02:44.830460 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.830221    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shcnr\" (UniqueName: \"kubernetes.io/projected/7480613f-9415-433c-b63c-72501451864d-kube-api-access-shcnr\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 23:02:44.995301 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.995269    2563 generic.go:356] "Generic (PLEG): container finished" podID="acc7d7c1-cb50-4705-9676-3fedeacdda8c" containerID="6f4df81472038ccee17dbd75e1b258d10f5bf3ca26fe17a6ce93f1c1aa717a12" exitCode=0
Jul 28 23:02:44.995522 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.995326    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-3682ae-7985768976-vtr77"
Jul 28 23:02:44.995522 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.995378    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-3682ae-7985768976-vtr77" event={"ID":"acc7d7c1-cb50-4705-9676-3fedeacdda8c","Type":"ContainerDied","Data":"6f4df81472038ccee17dbd75e1b258d10f5bf3ca26fe17a6ce93f1c1aa717a12"}
Jul 28 23:02:44.995522 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.995434    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-3682ae-7985768976-vtr77" event={"ID":"acc7d7c1-cb50-4705-9676-3fedeacdda8c","Type":"ContainerDied","Data":"6fcac7381c8ca55ce97dafd6f558b4b6f38eac43e5cfd9734370c02b7da049eb"}
Jul 28 23:02:44.995522 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.995459    2563 scope.go:117] "RemoveContainer" containerID="6f4df81472038ccee17dbd75e1b258d10f5bf3ca26fe17a6ce93f1c1aa717a12"
Jul 28 23:02:44.996747 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.996722    2563 generic.go:356] "Generic (PLEG): container finished" podID="7480613f-9415-433c-b63c-72501451864d" containerID="84b33b2e7c0da322da02d7fdcff6390ae5994b9d6c8de35ddee4053951d3070d" exitCode=0
Jul 28 23:02:44.996883 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.996752    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-3682ae-68d4966958-dnpv4"
Jul 28 23:02:44.996883 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.996812    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-3682ae-68d4966958-dnpv4" event={"ID":"7480613f-9415-433c-b63c-72501451864d","Type":"ContainerDied","Data":"84b33b2e7c0da322da02d7fdcff6390ae5994b9d6c8de35ddee4053951d3070d"}
Jul 28 23:02:44.996883 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.996866    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-3682ae-68d4966958-dnpv4" event={"ID":"7480613f-9415-433c-b63c-72501451864d","Type":"ContainerDied","Data":"5b22f5ea6beb90708103eaf15de1d2e993c7415902802d75963fb749c647717f"}
Jul 28 23:02:44.998312 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.998285    2563 generic.go:356] "Generic (PLEG): container finished" podID="0a2a293b-ce61-4e53-8821-79c92242ae07" containerID="b73573f908364ba30d1db57bf2e07abcfb79d63a0a2648aba9df64643023eb75" exitCode=0
Jul 28 23:02:44.998469 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.998341    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3682ae-5f995b88d9-vclmw"
Jul 28 23:02:44.998469 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.998409    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3682ae-5f995b88d9-vclmw" event={"ID":"0a2a293b-ce61-4e53-8821-79c92242ae07","Type":"ContainerDied","Data":"b73573f908364ba30d1db57bf2e07abcfb79d63a0a2648aba9df64643023eb75"}
Jul 28 23:02:44.998469 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:44.998450    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3682ae-5f995b88d9-vclmw" event={"ID":"0a2a293b-ce61-4e53-8821-79c92242ae07","Type":"ContainerDied","Data":"b925da4721e9c32f6073103a9ecf62d4d3174faac187b182ad02b4397244f978"}
Jul 28 23:02:45.005858 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:45.005837    2563 scope.go:117] "RemoveContainer" containerID="6f4df81472038ccee17dbd75e1b258d10f5bf3ca26fe17a6ce93f1c1aa717a12"
Jul 28 23:02:45.006173 ip-10-0-128-37 kubenswrapper[2563]: E0728 23:02:45.006149    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f4df81472038ccee17dbd75e1b258d10f5bf3ca26fe17a6ce93f1c1aa717a12\": container with ID starting with 6f4df81472038ccee17dbd75e1b258d10f5bf3ca26fe17a6ce93f1c1aa717a12 not found: ID does not exist" containerID="6f4df81472038ccee17dbd75e1b258d10f5bf3ca26fe17a6ce93f1c1aa717a12"
Jul 28 23:02:45.006309 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:45.006183    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f4df81472038ccee17dbd75e1b258d10f5bf3ca26fe17a6ce93f1c1aa717a12"} err="failed to get container status \"6f4df81472038ccee17dbd75e1b258d10f5bf3ca26fe17a6ce93f1c1aa717a12\": rpc error: code = NotFound desc = could not find container \"6f4df81472038ccee17dbd75e1b258d10f5bf3ca26fe17a6ce93f1c1aa717a12\": container with ID starting with 6f4df81472038ccee17dbd75e1b258d10f5bf3ca26fe17a6ce93f1c1aa717a12 not found: ID does not exist"
Jul 28 23:02:45.006309 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:45.006206    2563 scope.go:117] "RemoveContainer" containerID="84b33b2e7c0da322da02d7fdcff6390ae5994b9d6c8de35ddee4053951d3070d"
Jul 28 23:02:45.015345 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:45.015328    2563 scope.go:117] "RemoveContainer" containerID="84b33b2e7c0da322da02d7fdcff6390ae5994b9d6c8de35ddee4053951d3070d"
Jul 28 23:02:45.015720 ip-10-0-128-37 kubenswrapper[2563]: E0728 23:02:45.015695    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84b33b2e7c0da322da02d7fdcff6390ae5994b9d6c8de35ddee4053951d3070d\": container with ID starting with 84b33b2e7c0da322da02d7fdcff6390ae5994b9d6c8de35ddee4053951d3070d not found: ID does not exist" containerID="84b33b2e7c0da322da02d7fdcff6390ae5994b9d6c8de35ddee4053951d3070d"
Jul 28 23:02:45.015806 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:45.015722    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84b33b2e7c0da322da02d7fdcff6390ae5994b9d6c8de35ddee4053951d3070d"} err="failed to get container status \"84b33b2e7c0da322da02d7fdcff6390ae5994b9d6c8de35ddee4053951d3070d\": rpc error: code = NotFound desc = could not find container \"84b33b2e7c0da322da02d7fdcff6390ae5994b9d6c8de35ddee4053951d3070d\": container with ID starting with 84b33b2e7c0da322da02d7fdcff6390ae5994b9d6c8de35ddee4053951d3070d not found: ID does not exist"
Jul 28 23:02:45.015806 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:45.015739    2563 scope.go:117] "RemoveContainer" containerID="b73573f908364ba30d1db57bf2e07abcfb79d63a0a2648aba9df64643023eb75"
Jul 28 23:02:45.023583 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:45.023558    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-3682ae-68d4966958-dnpv4"]
Jul 28 23:02:45.023947 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:45.023930    2563 scope.go:117] "RemoveContainer" containerID="b73573f908364ba30d1db57bf2e07abcfb79d63a0a2648aba9df64643023eb75"
Jul 28 23:02:45.024168 ip-10-0-128-37 kubenswrapper[2563]: E0728 23:02:45.024145    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b73573f908364ba30d1db57bf2e07abcfb79d63a0a2648aba9df64643023eb75\": container with ID starting with b73573f908364ba30d1db57bf2e07abcfb79d63a0a2648aba9df64643023eb75 not found: ID does not exist" containerID="b73573f908364ba30d1db57bf2e07abcfb79d63a0a2648aba9df64643023eb75"
Jul 28 23:02:45.024220 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:45.024175    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b73573f908364ba30d1db57bf2e07abcfb79d63a0a2648aba9df64643023eb75"} err="failed to get container status \"b73573f908364ba30d1db57bf2e07abcfb79d63a0a2648aba9df64643023eb75\": rpc error: code = NotFound desc = could not find container \"b73573f908364ba30d1db57bf2e07abcfb79d63a0a2648aba9df64643023eb75\": container with ID starting with b73573f908364ba30d1db57bf2e07abcfb79d63a0a2648aba9df64643023eb75 not found: ID does not exist"
Jul 28 23:02:45.027516 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:45.027490    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-3682ae-68d4966958-dnpv4"]
Jul 28 23:02:45.037774 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:45.037751    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-3682ae-7985768976-vtr77"]
Jul 28 23:02:45.042849 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:45.042829    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-3682ae-7985768976-vtr77"]
Jul 28 23:02:45.059252 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:45.059229    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-3682ae-5f995b88d9-vclmw"]
Jul 28 23:02:45.064391 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:45.064372    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-3682ae-5f995b88d9-vclmw"]
Jul 28 23:02:45.130507 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:45.130479    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a2a293b-ce61-4e53-8821-79c92242ae07" path="/var/lib/kubelet/pods/0a2a293b-ce61-4e53-8821-79c92242ae07/volumes"
Jul 28 23:02:45.130858 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:45.130845    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7480613f-9415-433c-b63c-72501451864d" path="/var/lib/kubelet/pods/7480613f-9415-433c-b63c-72501451864d/volumes"
Jul 28 23:02:45.131165 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:45.131153    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acc7d7c1-cb50-4705-9676-3fedeacdda8c" path="/var/lib/kubelet/pods/acc7d7c1-cb50-4705-9676-3fedeacdda8c/volumes"
Jul 28 23:02:56.787541 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:56.787494    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-aa2de7-6d886fb56c-ggntt"]
Jul 28 23:02:56.788023 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:56.787777    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-aa2de7-6d886fb56c-ggntt" podUID="0668771a-becd-4b6c-b38e-36b46862a525" containerName="maas-api" containerID="cri-o://192e1bcf1a7a8aa4ec01015430210d7b7d73696ef05977de651965ac2a5e1981" gracePeriod=30
Jul 28 23:02:56.889559 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:56.889512    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-aa2de7-7f7cb8bfb-sr6jt"]
Jul 28 23:02:56.889789 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:56.889753    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-aa2de7-7f7cb8bfb-sr6jt" podUID="c0242d98-9729-4007-9293-c893ec1a40fc" containerName="payload-pre-processing" containerID="cri-o://383f16c71200b070bd039b910a399267ae97ae974184082955d4779f8019c8be" gracePeriod=30
Jul 28 23:02:57.061934 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:57.061896    2563 generic.go:356] "Generic (PLEG): container finished" podID="0668771a-becd-4b6c-b38e-36b46862a525" containerID="192e1bcf1a7a8aa4ec01015430210d7b7d73696ef05977de651965ac2a5e1981" exitCode=0
Jul 28 23:02:57.062126 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:57.062106    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-aa2de7-6d886fb56c-ggntt" event={"ID":"0668771a-becd-4b6c-b38e-36b46862a525","Type":"ContainerDied","Data":"192e1bcf1a7a8aa4ec01015430210d7b7d73696ef05977de651965ac2a5e1981"}
Jul 28 23:02:57.064569 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:57.064539    2563 generic.go:356] "Generic (PLEG): container finished" podID="c0242d98-9729-4007-9293-c893ec1a40fc" containerID="383f16c71200b070bd039b910a399267ae97ae974184082955d4779f8019c8be" exitCode=0
Jul 28 23:02:57.064703 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:57.064627    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-aa2de7-7f7cb8bfb-sr6jt" event={"ID":"c0242d98-9729-4007-9293-c893ec1a40fc","Type":"ContainerDied","Data":"383f16c71200b070bd039b910a399267ae97ae974184082955d4779f8019c8be"}
Jul 28 23:02:57.079216 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:57.079194    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-aa2de7-6d886fb56c-ggntt"
Jul 28 23:02:57.141497 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:57.141467    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqkpq\" (UniqueName: \"kubernetes.io/projected/0668771a-becd-4b6c-b38e-36b46862a525-kube-api-access-mqkpq\") pod \"0668771a-becd-4b6c-b38e-36b46862a525\" (UID: \"0668771a-becd-4b6c-b38e-36b46862a525\") "
Jul 28 23:02:57.141737 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:57.141522    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0668771a-becd-4b6c-b38e-36b46862a525-openshift-service-ca\") pod \"0668771a-becd-4b6c-b38e-36b46862a525\" (UID: \"0668771a-becd-4b6c-b38e-36b46862a525\") "
Jul 28 23:02:57.141737 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:57.141561    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0668771a-becd-4b6c-b38e-36b46862a525-maas-api-tls\") pod \"0668771a-becd-4b6c-b38e-36b46862a525\" (UID: \"0668771a-becd-4b6c-b38e-36b46862a525\") "
Jul 28 23:02:57.142150 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:57.142122    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0668771a-becd-4b6c-b38e-36b46862a525-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0668771a-becd-4b6c-b38e-36b46862a525" (UID: "0668771a-becd-4b6c-b38e-36b46862a525"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:02:57.143865 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:57.143831    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0668771a-becd-4b6c-b38e-36b46862a525-kube-api-access-mqkpq" (OuterVolumeSpecName: "kube-api-access-mqkpq") pod "0668771a-becd-4b6c-b38e-36b46862a525" (UID: "0668771a-becd-4b6c-b38e-36b46862a525"). InnerVolumeSpecName "kube-api-access-mqkpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 23:02:57.144139 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:57.144116    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0668771a-becd-4b6c-b38e-36b46862a525-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0668771a-becd-4b6c-b38e-36b46862a525" (UID: "0668771a-becd-4b6c-b38e-36b46862a525"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 23:02:57.162981 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:57.162957    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-aa2de7-7f7cb8bfb-sr6jt"
Jul 28 23:02:57.243176 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:57.243143    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0242d98-9729-4007-9293-c893ec1a40fc-plugins-config-volume\") pod \"c0242d98-9729-4007-9293-c893ec1a40fc\" (UID: \"c0242d98-9729-4007-9293-c893ec1a40fc\") "
Jul 28 23:02:57.243326 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:57.243265    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9whpx\" (UniqueName: \"kubernetes.io/projected/c0242d98-9729-4007-9293-c893ec1a40fc-kube-api-access-9whpx\") pod \"c0242d98-9729-4007-9293-c893ec1a40fc\" (UID: \"c0242d98-9729-4007-9293-c893ec1a40fc\") "
Jul 28 23:02:57.243548 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:57.243531    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqkpq\" (UniqueName: \"kubernetes.io/projected/0668771a-becd-4b6c-b38e-36b46862a525-kube-api-access-mqkpq\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 23:02:57.243548 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:57.243537    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0242d98-9729-4007-9293-c893ec1a40fc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c0242d98-9729-4007-9293-c893ec1a40fc" (UID: "c0242d98-9729-4007-9293-c893ec1a40fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:02:57.243650 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:57.243554    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0668771a-becd-4b6c-b38e-36b46862a525-openshift-service-ca\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 23:02:57.243650 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:57.243569    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0668771a-becd-4b6c-b38e-36b46862a525-maas-api-tls\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 23:02:57.245293 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:57.245264    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0242d98-9729-4007-9293-c893ec1a40fc-kube-api-access-9whpx" (OuterVolumeSpecName: "kube-api-access-9whpx") pod "c0242d98-9729-4007-9293-c893ec1a40fc" (UID: "c0242d98-9729-4007-9293-c893ec1a40fc"). InnerVolumeSpecName "kube-api-access-9whpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 23:02:57.344761 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:57.344683    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0242d98-9729-4007-9293-c893ec1a40fc-plugins-config-volume\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 23:02:57.344761 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:57.344711    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9whpx\" (UniqueName: \"kubernetes.io/projected/c0242d98-9729-4007-9293-c893ec1a40fc-kube-api-access-9whpx\") on node \"ip-10-0-128-37.ec2.internal\" DevicePath \"\""
Jul 28 23:02:58.071887 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:58.071799    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-aa2de7-7f7cb8bfb-sr6jt"
Jul 28 23:02:58.071887 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:58.071819    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-aa2de7-7f7cb8bfb-sr6jt" event={"ID":"c0242d98-9729-4007-9293-c893ec1a40fc","Type":"ContainerDied","Data":"239d1c13e5198cbb6b60cb8aa4b66c1763a9736da4040be5a5b53ed72774d5e1"}
Jul 28 23:02:58.072451 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:58.071898    2563 scope.go:117] "RemoveContainer" containerID="383f16c71200b070bd039b910a399267ae97ae974184082955d4779f8019c8be"
Jul 28 23:02:58.073328 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:58.073312    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-aa2de7-6d886fb56c-ggntt"
Jul 28 23:02:58.073451 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:58.073324    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-aa2de7-6d886fb56c-ggntt" event={"ID":"0668771a-becd-4b6c-b38e-36b46862a525","Type":"ContainerDied","Data":"64e903ea71eb53bea4db9514bfe553539059608ce6a21461af9f9caebc1b5ef2"}
Jul 28 23:02:58.080677 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:58.080655    2563 scope.go:117] "RemoveContainer" containerID="192e1bcf1a7a8aa4ec01015430210d7b7d73696ef05977de651965ac2a5e1981"
Jul 28 23:02:58.097229 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:58.097209    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-aa2de7-7f7cb8bfb-sr6jt"]
Jul 28 23:02:58.100979 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:58.100956    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-aa2de7-7f7cb8bfb-sr6jt"]
Jul 28 23:02:58.111483 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:58.111458    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-aa2de7-6d886fb56c-ggntt"]
Jul 28 23:02:58.113031 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:58.113013    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-aa2de7-6d886fb56c-ggntt"]
Jul 28 23:02:59.129845 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:59.129813    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0668771a-becd-4b6c-b38e-36b46862a525" path="/var/lib/kubelet/pods/0668771a-becd-4b6c-b38e-36b46862a525/volumes"
Jul 28 23:02:59.130237 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:02:59.130163    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0242d98-9729-4007-9293-c893ec1a40fc" path="/var/lib/kubelet/pods/c0242d98-9729-4007-9293-c893ec1a40fc/volumes"
Jul 28 23:03:10.219977 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:10.219943    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8687496d99-crms2_07403f33-2401-479b-8b78-3b0c0b2716e9/manager/0.log"
Jul 28 23:03:10.345017 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:10.344971    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-mwl94_9072fe04-b34c-4640-81ba-da5e1b87134a/server/0.log"
Jul 28 23:03:10.609105 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:10.609023    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6974ffcb4c-bsd4p_dfe553a9-7cf9-4715-865d-77d843be49a3/manager/0.log"
Jul 28 23:03:10.615871 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:10.615850    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6974ffcb4c-bsd4p_dfe553a9-7cf9-4715-865d-77d843be49a3/copy-manifests/0.log"
Jul 28 23:03:11.231716 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:11.231680    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-fcfcd7d96-cpvhs_1b67e040-3901-4cf0-bd2d-3224954ba70e/maas-api/0.log"
Jul 28 23:03:11.717554 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:11.717524    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-h7cc5_07f92c4d-16a3-41de-8d51-ecfa6e5a32b5/postgres/0.log"
Jul 28 23:03:13.177689 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:13.177654    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-7rkst_01e11fb0-1943-4fd6-8beb-1146b9723726/manager/0.log"
Jul 28 23:03:13.523373 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:13.523329    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-jlv6n_be85f180-b55d-42d8-931c-fdef76639076/registry-server/0.log"
Jul 28 23:03:13.646145 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:13.646114    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-6hfw5_7eb29594-0911-45bb-bf64-52eb49612f7d/manager/0.log"
Jul 28 23:03:14.376347 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:14.376295    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-fn9t5_12c068fd-02c5-4ff8-8ec5-fe35c37d3139/discovery/0.log"
Jul 28 23:03:14.501577 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:14.501527    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9899fcb89-58jft_da9ae5dc-c3b4-459b-a25f-76b16e4ea340/kube-auth-proxy/0.log"
Jul 28 23:03:14.879131 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:14.879100    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-2zd9d_b164ba31-adb2-42c8-9704-d398ba2650eb/payload-pre-processing/0.log"
Jul 28 23:03:15.486794 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:15.486765    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-xbrnq_7f8f63be-a0e6-4b21-912d-839301d1bf0c/main/0.log"
Jul 28 23:03:15.742024 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:15.741955    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-lppq7_acd46159-9704-48f8-b0ec-a5855c2c804a/main/0.log"
Jul 28 23:03:15.983632 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:15.983597    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-svj7p_862c1e2f-c706-4da3-a357-530decd9458a/main/0.log"
Jul 28 23:03:24.363278 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:24.363239    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2rqhg_5d7dfdb2-a77a-47e8-865a-07a014ec5633/global-pull-secret-syncer/0.log"
Jul 28 23:03:24.577947 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:24.574968    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7wmm5_6f95fc9f-b8c8-433e-ae9b-e20fcd167fbe/konnectivity-agent/0.log"
Jul 28 23:03:24.595652 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:24.595620    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-37.ec2.internal_2aaf76ae458a97a3e5edaa3e36196e52/haproxy/0.log"
Jul 28 23:03:29.872610 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:29.872578    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-7rkst_01e11fb0-1943-4fd6-8beb-1146b9723726/manager/0.log"
Jul 28 23:03:29.975679 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:29.975646    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-jlv6n_be85f180-b55d-42d8-931c-fdef76639076/registry-server/0.log"
Jul 28 23:03:30.164875 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:30.164790    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-6hfw5_7eb29594-0911-45bb-bf64-52eb49612f7d/manager/0.log"
Jul 28 23:03:32.189469 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:32.189441    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-77cbc6f9d4-hxcn7_42c93fdc-261d-46e3-82f6-89d82c18df1b/metrics-server/0.log"
Jul 28 23:03:32.328197 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:32.328171    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dpmd2_59091947-e558-4909-8c6a-85a008823b1c/node-exporter/0.log"
Jul 28 23:03:32.354496 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:32.354471    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dpmd2_59091947-e558-4909-8c6a-85a008823b1c/kube-rbac-proxy/0.log"
Jul 28 23:03:32.376951 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:32.376928    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dpmd2_59091947-e558-4909-8c6a-85a008823b1c/init-textfile/0.log"
Jul 28 23:03:32.760397 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:32.760368    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-fcjgp_3d007a2c-32b5-4539-9b07-4b4a27973482/prometheus-operator/0.log"
Jul 28 23:03:32.783446 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:32.783421    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-fcjgp_3d007a2c-32b5-4539-9b07-4b4a27973482/kube-rbac-proxy/0.log"
Jul 28 23:03:32.806146 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:32.806120    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-lwvw5_85f15809-aac0-4ccb-a862-7d1ea31d353b/prometheus-operator-admission-webhook/0.log"
Jul 28 23:03:34.640416 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.640377    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-x8jld/perf-node-gather-daemonset-4fxxl"]
Jul 28 23:03:34.640822 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.640712    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d61132a7-a786-4441-8c5c-c31fcea80679" containerName="payload-pre-processing"
Jul 28 23:03:34.640822 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.640724    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d61132a7-a786-4441-8c5c-c31fcea80679" containerName="payload-pre-processing"
Jul 28 23:03:34.640822 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.640738    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7480613f-9415-433c-b63c-72501451864d" containerName="payload-processing"
Jul 28 23:03:34.640822 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.640743    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7480613f-9415-433c-b63c-72501451864d" containerName="payload-processing"
Jul 28 23:03:34.640822 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.640750    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a2a293b-ce61-4e53-8821-79c92242ae07" containerName="maas-api"
Jul 28 23:03:34.640822 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.640756    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a2a293b-ce61-4e53-8821-79c92242ae07" containerName="maas-api"
Jul 28 23:03:34.640822 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.640761    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acc7d7c1-cb50-4705-9676-3fedeacdda8c" containerName="payload-pre-processing"
Jul 28 23:03:34.640822 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.640766    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="acc7d7c1-cb50-4705-9676-3fedeacdda8c" containerName="payload-pre-processing"
Jul 28 23:03:34.640822 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.640773    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0242d98-9729-4007-9293-c893ec1a40fc" containerName="payload-pre-processing"
Jul 28 23:03:34.640822 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.640778    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0242d98-9729-4007-9293-c893ec1a40fc" containerName="payload-pre-processing"
Jul 28 23:03:34.640822 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.640793    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0668771a-becd-4b6c-b38e-36b46862a525" containerName="maas-api"
Jul 28 23:03:34.640822 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.640798    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0668771a-becd-4b6c-b38e-36b46862a525" containerName="maas-api"
Jul 28 23:03:34.640822 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.640804    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aede49d6-041a-48c4-bd47-8364a383a641" containerName="payload-pre-processing"
Jul 28 23:03:34.640822 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.640809    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="aede49d6-041a-48c4-bd47-8364a383a641" containerName="payload-pre-processing"
Jul 28 23:03:34.640822 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.640815    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da61fa5c-8cd3-4869-9abd-c53c74ce93d5" containerName="maas-api"
Jul 28 23:03:34.640822 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.640821    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="da61fa5c-8cd3-4869-9abd-c53c74ce93d5" containerName="maas-api"
Jul 28 23:03:34.640822 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.640826    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16f9d31b-a579-4187-8d9d-6a0bbf2919f7" containerName="payload-processing"
Jul 28 23:03:34.640822 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.640832    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="16f9d31b-a579-4187-8d9d-6a0bbf2919f7" containerName="payload-processing"
Jul 28 23:03:34.641747 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.640837    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a" containerName="maas-api"
Jul 28 23:03:34.641747 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.640843    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a" containerName="maas-api"
Jul 28 23:03:34.641747 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.640850    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3670d0d-3532-4f09-9804-e65ad2ea648c" containerName="payload-pre-processing"
Jul 28 23:03:34.641747 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.640855    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3670d0d-3532-4f09-9804-e65ad2ea648c" containerName="payload-pre-processing"
Jul 28 23:03:34.641747 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.640916    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7480613f-9415-433c-b63c-72501451864d" containerName="payload-processing"
Jul 28 23:03:34.641747 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.640923    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="aede49d6-041a-48c4-bd47-8364a383a641" containerName="payload-pre-processing"
Jul 28 23:03:34.641747 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.640930    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0242d98-9729-4007-9293-c893ec1a40fc" containerName="payload-pre-processing"
Jul 28 23:03:34.641747 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.640936    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3670d0d-3532-4f09-9804-e65ad2ea648c" containerName="payload-pre-processing"
Jul 28 23:03:34.641747 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.640942    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a2a293b-ce61-4e53-8821-79c92242ae07" containerName="maas-api"
Jul 28 23:03:34.641747 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.640947    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ffedae5-fde3-4a8d-90e2-4fb81da7aa2a" containerName="maas-api"
Jul 28 23:03:34.641747 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.640954    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="16f9d31b-a579-4187-8d9d-6a0bbf2919f7" containerName="payload-processing"
Jul 28 23:03:34.641747 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.640959    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="da61fa5c-8cd3-4869-9abd-c53c74ce93d5" containerName="maas-api"
Jul 28 23:03:34.641747 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.640966    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0668771a-becd-4b6c-b38e-36b46862a525" containerName="maas-api"
Jul 28 23:03:34.641747 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.640971    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d61132a7-a786-4441-8c5c-c31fcea80679" containerName="payload-pre-processing"
Jul 28 23:03:34.641747 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.640977    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="acc7d7c1-cb50-4705-9676-3fedeacdda8c" containerName="payload-pre-processing"
Jul 28 23:03:34.643964 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.643946    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x8jld/perf-node-gather-daemonset-4fxxl"
Jul 28 23:03:34.646604 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.646580    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x8jld\"/\"kube-root-ca.crt\""
Jul 28 23:03:34.647763 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.647743    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x8jld\"/\"openshift-service-ca.crt\""
Jul 28 23:03:34.647874 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.647784    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-x8jld\"/\"default-dockercfg-jw6hm\""
Jul 28 23:03:34.653062 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.653037    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x8jld/perf-node-gather-daemonset-4fxxl"]
Jul 28 23:03:34.673061 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.673033    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ca0e2c96-d8ca-49f1-8e35-3bf60625a3af-proc\") pod \"perf-node-gather-daemonset-4fxxl\" (UID: \"ca0e2c96-d8ca-49f1-8e35-3bf60625a3af\") " pod="openshift-must-gather-x8jld/perf-node-gather-daemonset-4fxxl"
Jul 28 23:03:34.673175 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.673069    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ca0e2c96-d8ca-49f1-8e35-3bf60625a3af-lib-modules\") pod \"perf-node-gather-daemonset-4fxxl\" (UID: \"ca0e2c96-d8ca-49f1-8e35-3bf60625a3af\") " pod="openshift-must-gather-x8jld/perf-node-gather-daemonset-4fxxl"
Jul 28 23:03:34.673175 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.673092    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ca0e2c96-d8ca-49f1-8e35-3bf60625a3af-podres\") pod \"perf-node-gather-daemonset-4fxxl\" (UID: \"ca0e2c96-d8ca-49f1-8e35-3bf60625a3af\") " pod="openshift-must-gather-x8jld/perf-node-gather-daemonset-4fxxl"
Jul 28 23:03:34.673175 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.673159    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj58p\" (UniqueName: \"kubernetes.io/projected/ca0e2c96-d8ca-49f1-8e35-3bf60625a3af-kube-api-access-xj58p\") pod \"perf-node-gather-daemonset-4fxxl\" (UID: \"ca0e2c96-d8ca-49f1-8e35-3bf60625a3af\") " pod="openshift-must-gather-x8jld/perf-node-gather-daemonset-4fxxl"
Jul 28 23:03:34.673287 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.673197    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca0e2c96-d8ca-49f1-8e35-3bf60625a3af-sys\") pod \"perf-node-gather-daemonset-4fxxl\" (UID: \"ca0e2c96-d8ca-49f1-8e35-3bf60625a3af\") " pod="openshift-must-gather-x8jld/perf-node-gather-daemonset-4fxxl"
Jul 28 23:03:34.773943 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.773911    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca0e2c96-d8ca-49f1-8e35-3bf60625a3af-sys\") pod \"perf-node-gather-daemonset-4fxxl\" (UID: \"ca0e2c96-d8ca-49f1-8e35-3bf60625a3af\") " pod="openshift-must-gather-x8jld/perf-node-gather-daemonset-4fxxl"
Jul 28 23:03:34.774095 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.773975    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ca0e2c96-d8ca-49f1-8e35-3bf60625a3af-proc\") pod \"perf-node-gather-daemonset-4fxxl\" (UID: \"ca0e2c96-d8ca-49f1-8e35-3bf60625a3af\") " pod="openshift-must-gather-x8jld/perf-node-gather-daemonset-4fxxl"
Jul 28 23:03:34.774095 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.774008    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ca0e2c96-d8ca-49f1-8e35-3bf60625a3af-lib-modules\") pod \"perf-node-gather-daemonset-4fxxl\" (UID: \"ca0e2c96-d8ca-49f1-8e35-3bf60625a3af\") " pod="openshift-must-gather-x8jld/perf-node-gather-daemonset-4fxxl"
Jul 28 23:03:34.774095 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.774029    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ca0e2c96-d8ca-49f1-8e35-3bf60625a3af-podres\") pod \"perf-node-gather-daemonset-4fxxl\" (UID: \"ca0e2c96-d8ca-49f1-8e35-3bf60625a3af\") " pod="openshift-must-gather-x8jld/perf-node-gather-daemonset-4fxxl"
Jul 28 23:03:34.774095 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.774040    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca0e2c96-d8ca-49f1-8e35-3bf60625a3af-sys\") pod \"perf-node-gather-daemonset-4fxxl\" (UID: \"ca0e2c96-d8ca-49f1-8e35-3bf60625a3af\") " pod="openshift-must-gather-x8jld/perf-node-gather-daemonset-4fxxl"
Jul 28 23:03:34.774095 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.774061    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj58p\" (UniqueName: \"kubernetes.io/projected/ca0e2c96-d8ca-49f1-8e35-3bf60625a3af-kube-api-access-xj58p\") pod \"perf-node-gather-daemonset-4fxxl\" (UID: \"ca0e2c96-d8ca-49f1-8e35-3bf60625a3af\") " pod="openshift-must-gather-x8jld/perf-node-gather-daemonset-4fxxl"
Jul 28 23:03:34.774277 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.774114    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ca0e2c96-d8ca-49f1-8e35-3bf60625a3af-proc\") pod \"perf-node-gather-daemonset-4fxxl\" (UID: \"ca0e2c96-d8ca-49f1-8e35-3bf60625a3af\") " pod="openshift-must-gather-x8jld/perf-node-gather-daemonset-4fxxl"
Jul 28 23:03:34.774277 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.774145    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ca0e2c96-d8ca-49f1-8e35-3bf60625a3af-podres\") pod \"perf-node-gather-daemonset-4fxxl\" (UID: \"ca0e2c96-d8ca-49f1-8e35-3bf60625a3af\") " pod="openshift-must-gather-x8jld/perf-node-gather-daemonset-4fxxl"
Jul 28 23:03:34.774277 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.774162    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ca0e2c96-d8ca-49f1-8e35-3bf60625a3af-lib-modules\") pod \"perf-node-gather-daemonset-4fxxl\" (UID: \"ca0e2c96-d8ca-49f1-8e35-3bf60625a3af\") " pod="openshift-must-gather-x8jld/perf-node-gather-daemonset-4fxxl"
Jul 28 23:03:34.781901 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.781880    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj58p\" (UniqueName: \"kubernetes.io/projected/ca0e2c96-d8ca-49f1-8e35-3bf60625a3af-kube-api-access-xj58p\") pod \"perf-node-gather-daemonset-4fxxl\" (UID: \"ca0e2c96-d8ca-49f1-8e35-3bf60625a3af\") " pod="openshift-must-gather-x8jld/perf-node-gather-daemonset-4fxxl"
Jul 28 23:03:34.954066 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:34.954038    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x8jld/perf-node-gather-daemonset-4fxxl"
Jul 28 23:03:35.101308 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:35.101274    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x8jld/perf-node-gather-daemonset-4fxxl"]
Jul 28 23:03:35.103376 ip-10-0-128-37 kubenswrapper[2563]: W0728 23:03:35.103333    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podca0e2c96_d8ca_49f1_8e35_3bf60625a3af.slice/crio-d3db487614d84ce0319626b797f02ba6ae20789289c7cb63486f88e789f99e3c WatchSource:0}: Error finding container d3db487614d84ce0319626b797f02ba6ae20789289c7cb63486f88e789f99e3c: Status 404 returned error can't find the container with id d3db487614d84ce0319626b797f02ba6ae20789289c7cb63486f88e789f99e3c
Jul 28 23:03:35.244875 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:35.244849    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x8jld/perf-node-gather-daemonset-4fxxl" event={"ID":"ca0e2c96-d8ca-49f1-8e35-3bf60625a3af","Type":"ContainerStarted","Data":"d3db487614d84ce0319626b797f02ba6ae20789289c7cb63486f88e789f99e3c"}
Jul 28 23:03:35.457969 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:35.457942    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-dd875b845-229jm_3757325f-8310-4c4d-bf0a-8686acf3c88c/console/0.log"
Jul 28 23:03:36.250607 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:36.250563    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x8jld/perf-node-gather-daemonset-4fxxl" event={"ID":"ca0e2c96-d8ca-49f1-8e35-3bf60625a3af","Type":"ContainerStarted","Data":"cce96c2ead85c81f3138f8bcc6f057a31235e9d154c5a01f2d3a57410ac56f69"}
Jul 28 23:03:36.251018 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:36.250709    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-x8jld/perf-node-gather-daemonset-4fxxl"
Jul 28 23:03:36.265814 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:36.265766    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-x8jld/perf-node-gather-daemonset-4fxxl" podStartSLOduration=2.265751663 podStartE2EDuration="2.265751663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 23:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:03:36.26458071 +0000 UTC m=+3513.644718354" watchObservedRunningTime="2026-07-28 23:03:36.265751663 +0000 UTC m=+3513.645889306"
Jul 28 23:03:37.156475 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:37.156443    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vqx8j_331ef64d-2011-4149-b38c-88921f08f078/dns/0.log"
Jul 28 23:03:37.179030 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:37.179002    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vqx8j_331ef64d-2011-4149-b38c-88921f08f078/kube-rbac-proxy/0.log"
Jul 28 23:03:37.256451 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:37.256423    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qtgkr_820ee419-dab7-43ef-94bf-cadbaf9a5232/dns-node-resolver/0.log"
Jul 28 23:03:37.779682 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:37.779649    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-86664d9944-5hsqd_bb491a55-2f3a-4096-a4b1-e09b62d3bef9/registry/0.log"
Jul 28 23:03:37.853332 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:37.853305    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tx5m6_2a9cac08-b862-4f7b-88e3-f4de98a035ea/node-ca/0.log"
Jul 28 23:03:38.903091 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:38.903058    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-fn9t5_12c068fd-02c5-4ff8-8ec5-fe35c37d3139/discovery/0.log"
Jul 28 23:03:38.923718 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:38.923692    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9899fcb89-58jft_da9ae5dc-c3b4-459b-a25f-76b16e4ea340/kube-auth-proxy/0.log"
Jul 28 23:03:39.063704 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:39.063677    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-2zd9d_b164ba31-adb2-42c8-9704-d398ba2650eb/payload-pre-processing/0.log"
Jul 28 23:03:39.973308 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:39.973267    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ssgsl_d653889b-b9a3-4d5d-bbf0-f286413a20c9/serve-healthcheck-canary/0.log"
Jul 28 23:03:40.547701 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:40.547675    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bdmbq_f91a796a-dc22-4189-bb32-f78b985973c6/kube-rbac-proxy/0.log"
Jul 28 23:03:40.570933 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:40.570906    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bdmbq_f91a796a-dc22-4189-bb32-f78b985973c6/exporter/0.log"
Jul 28 23:03:40.595193 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:40.595172    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bdmbq_f91a796a-dc22-4189-bb32-f78b985973c6/extractor/0.log"
Jul 28 23:03:42.264485 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:42.264457    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-x8jld/perf-node-gather-daemonset-4fxxl"
Jul 28 23:03:43.308847 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:43.308817    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8687496d99-crms2_07403f33-2401-479b-8b78-3b0c0b2716e9/manager/0.log"
Jul 28 23:03:43.329139 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:43.329114    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-mwl94_9072fe04-b34c-4640-81ba-da5e1b87134a/server/0.log"
Jul 28 23:03:43.465992 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:43.465968    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6974ffcb4c-bsd4p_dfe553a9-7cf9-4715-865d-77d843be49a3/manager/0.log"
Jul 28 23:03:43.486143 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:43.486122    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6974ffcb4c-bsd4p_dfe553a9-7cf9-4715-865d-77d843be49a3/copy-manifests/0.log"
Jul 28 23:03:45.080321 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:45.080292    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-fn9t5_12c068fd-02c5-4ff8-8ec5-fe35c37d3139/discovery/0.log"
Jul 28 23:03:45.090941 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:45.090914    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9899fcb89-58jft_da9ae5dc-c3b4-459b-a25f-76b16e4ea340/kube-auth-proxy/0.log"
Jul 28 23:03:45.158096 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:45.158066    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-pmlf6_f88674d2-586a-4fd9-ac95-711f572c5dc1/openshift-lws-operator/0.log"
Jul 28 23:03:45.192822 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:45.192790    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-2zd9d_b164ba31-adb2-42c8-9704-d398ba2650eb/payload-pre-processing/0.log"
Jul 28 23:03:46.810192 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:46.810159    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-fn9t5_12c068fd-02c5-4ff8-8ec5-fe35c37d3139/discovery/0.log"
Jul 28 23:03:46.824495 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:46.824473    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9899fcb89-58jft_da9ae5dc-c3b4-459b-a25f-76b16e4ea340/kube-auth-proxy/0.log"
Jul 28 23:03:46.939058 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:46.939026    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-2zd9d_b164ba31-adb2-42c8-9704-d398ba2650eb/payload-pre-processing/0.log"
Jul 28 23:03:51.554981 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:51.554951    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4fjz6_6eea54c2-c579-4dab-8222-909d88ccf205/kube-multus-additional-cni-plugins/0.log"
Jul 28 23:03:51.577576 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:51.577546    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4fjz6_6eea54c2-c579-4dab-8222-909d88ccf205/egress-router-binary-copy/0.log"
Jul 28 23:03:51.598303 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:51.598280    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4fjz6_6eea54c2-c579-4dab-8222-909d88ccf205/cni-plugins/0.log"
Jul 28 23:03:51.619421 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:51.619401    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4fjz6_6eea54c2-c579-4dab-8222-909d88ccf205/bond-cni-plugin/0.log"
Jul 28 23:03:51.643709 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:51.643689    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4fjz6_6eea54c2-c579-4dab-8222-909d88ccf205/routeoverride-cni/0.log"
Jul 28 23:03:51.665335 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:51.665316    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4fjz6_6eea54c2-c579-4dab-8222-909d88ccf205/whereabouts-cni-bincopy/0.log"
Jul 28 23:03:51.686961 ip-10-0-128-37 kubenswrapper[2563]: I0728 23:03:51.686941    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4fjz6_6eea54c2-c579-4dab-8222-909d88ccf205/whereabouts-cni/0.log"