Jul 24 16:42:57.231260 ip-10-0-133-234 systemd[1]: Starting Kubernetes Kubelet...
Jul 24 16:42:57.675580 ip-10-0-133-234 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 16:42:57.675580 ip-10-0-133-234 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 24 16:42:57.675580 ip-10-0-133-234 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 16:42:57.675580 ip-10-0-133-234 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 24 16:42:57.675580 ip-10-0-133-234 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 16:42:57.677652 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.677566    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 24 16:42:57.683193 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683172    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 16:42:57.683193 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683189    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 16:42:57.683193 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683195    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 16:42:57.683193 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683199    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 16:42:57.683445 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683205    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 16:42:57.683445 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683209    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 16:42:57.683445 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683213    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 16:42:57.683445 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683219    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 16:42:57.683445 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683225    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 16:42:57.683445 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683229    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 16:42:57.683445 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683233    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 16:42:57.683445 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683237    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 16:42:57.683445 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683241    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 16:42:57.683445 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683245    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 16:42:57.683445 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683248    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 16:42:57.683445 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683252    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 16:42:57.683445 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683256    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 16:42:57.683445 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683260    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 16:42:57.683445 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683265    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 16:42:57.683445 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683269    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 16:42:57.683445 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683273    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 16:42:57.683445 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683278    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 16:42:57.683445 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683281    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 16:42:57.684266 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683285    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 16:42:57.684266 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683289    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 16:42:57.684266 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683293    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 16:42:57.684266 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683298    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 16:42:57.684266 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683305    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 16:42:57.684266 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683311    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 16:42:57.684266 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683316    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 16:42:57.684266 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683321    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 16:42:57.684266 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683325    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 16:42:57.684266 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683330    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 16:42:57.684266 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683335    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 16:42:57.684266 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683339    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 16:42:57.684266 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683344    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 16:42:57.684266 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683348    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 16:42:57.684266 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683353    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 16:42:57.684266 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683357    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 16:42:57.684266 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683361    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 16:42:57.684266 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683366    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 16:42:57.684266 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683370    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 16:42:57.685099 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683374    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 16:42:57.685099 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683379    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 16:42:57.685099 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683383    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 16:42:57.685099 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683387    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 16:42:57.685099 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683391    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 16:42:57.685099 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683395    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 16:42:57.685099 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683399    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 16:42:57.685099 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683403    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 16:42:57.685099 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683407    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 16:42:57.685099 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683411    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 16:42:57.685099 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683415    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 16:42:57.685099 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683419    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 16:42:57.685099 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683424    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 16:42:57.685099 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683428    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 16:42:57.685099 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683432    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 16:42:57.685099 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683437    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 16:42:57.685099 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683441    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 16:42:57.685099 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683444    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 16:42:57.685099 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683449    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 16:42:57.685099 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683454    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 16:42:57.686026 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683458    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 16:42:57.686026 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683462    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 16:42:57.686026 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683466    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 16:42:57.686026 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683470    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 16:42:57.686026 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683475    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 16:42:57.686026 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683480    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 16:42:57.686026 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683484    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 16:42:57.686026 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683489    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 16:42:57.686026 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683493    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 16:42:57.686026 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683503    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 16:42:57.686026 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683507    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 16:42:57.686026 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683511    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 16:42:57.686026 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683515    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 16:42:57.686026 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683520    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 16:42:57.686026 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683524    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 16:42:57.686026 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683528    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 16:42:57.686026 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683534    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 16:42:57.686026 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683538    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 16:42:57.686026 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683543    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 16:42:57.686026 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683547    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 16:42:57.686893 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683551    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 16:42:57.686893 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683555    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 16:42:57.686893 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683559    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 16:42:57.686893 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683564    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 16:42:57.686893 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683568    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 16:42:57.686893 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683572    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 16:42:57.686893 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683576    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 16:42:57.686893 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683581    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 16:42:57.686893 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683585    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 16:42:57.686893 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683591    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 16:42:57.686893 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683597    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 16:42:57.686893 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683603    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 16:42:57.686893 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683607    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 16:42:57.686893 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683612    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 16:42:57.686893 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683617    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 16:42:57.686893 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683622    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 16:42:57.686893 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.683627    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 16:42:57.686893 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684776    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 16:42:57.686893 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684786    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 16:42:57.687743 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684792    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 16:42:57.687743 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684797    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 16:42:57.687743 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684802    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 16:42:57.687743 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684808    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 16:42:57.687743 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684812    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 16:42:57.687743 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684818    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 16:42:57.687743 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684823    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 16:42:57.687743 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684828    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 16:42:57.687743 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684832    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 16:42:57.687743 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684837    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 16:42:57.687743 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684841    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 16:42:57.687743 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684845    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 16:42:57.687743 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684849    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 16:42:57.687743 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684853    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 16:42:57.687743 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684857    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 16:42:57.687743 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684861    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 16:42:57.687743 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684866    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 16:42:57.687743 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684870    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 16:42:57.687743 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684874    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 16:42:57.687743 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684878    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 16:42:57.688354 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684883    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 16:42:57.688354 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684889    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 16:42:57.688354 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684894    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 16:42:57.688354 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684899    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 16:42:57.688354 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684904    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 16:42:57.688354 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684908    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 16:42:57.688354 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684915    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 16:42:57.688354 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684919    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 16:42:57.688354 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684923    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 16:42:57.688354 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684927    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 16:42:57.688354 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684953    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 16:42:57.688354 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684960    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 16:42:57.688354 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684966    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 16:42:57.688354 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684972    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 16:42:57.688354 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684977    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 16:42:57.688354 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684981    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 16:42:57.688354 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684987    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 16:42:57.688354 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684991    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 16:42:57.688354 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.684996    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 16:42:57.688835 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685000    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 16:42:57.688835 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685004    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 16:42:57.688835 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685008    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 16:42:57.688835 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685013    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 16:42:57.688835 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685017    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 16:42:57.688835 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685021    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 16:42:57.688835 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685025    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 16:42:57.688835 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685029    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 16:42:57.688835 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685033    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 16:42:57.688835 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685038    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 16:42:57.688835 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685042    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 16:42:57.688835 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685046    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 16:42:57.688835 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685050    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 16:42:57.688835 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685054    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 16:42:57.688835 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685058    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 16:42:57.688835 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685063    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 16:42:57.688835 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685067    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 16:42:57.688835 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685071    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 16:42:57.688835 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685076    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 16:42:57.688835 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685080    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 16:42:57.689431 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685087    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 16:42:57.689431 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685091    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 16:42:57.689431 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685095    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 16:42:57.689431 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685099    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 16:42:57.689431 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685102    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 16:42:57.689431 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685106    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 16:42:57.689431 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685110    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 16:42:57.689431 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685114    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 16:42:57.689431 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685118    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 16:42:57.689431 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685122    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 16:42:57.689431 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685128    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 16:42:57.689431 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685132    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 16:42:57.689431 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685136    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 16:42:57.689431 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685141    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 16:42:57.689431 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685146    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 16:42:57.689431 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685151    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 16:42:57.689431 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685155    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 16:42:57.689431 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685159    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 16:42:57.689431 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685163    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 16:42:57.689431 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685167    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 16:42:57.690062 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685171    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 16:42:57.690062 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685175    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 16:42:57.690062 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685180    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 16:42:57.690062 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685184    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 16:42:57.690062 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685188    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 16:42:57.690062 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685192    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 16:42:57.690062 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685196    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 16:42:57.690062 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685200    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 16:42:57.690062 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685205    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 16:42:57.690062 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685210    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 16:42:57.690062 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685213    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 16:42:57.690062 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685218    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 16:42:57.690062 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685225    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 16:42:57.690062 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685229    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 16:42:57.690062 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685233    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 16:42:57.690062 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685237    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 16:42:57.690062 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685242    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 16:42:57.690062 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685246    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 16:42:57.690062 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685854    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 16:42:57.690062 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685862    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 16:42:57.690691 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685867    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 16:42:57.690691 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685871    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 16:42:57.690691 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685877    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 16:42:57.690691 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685881    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 16:42:57.690691 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685886    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 16:42:57.690691 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685890    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 16:42:57.690691 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685894    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 16:42:57.690691 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685899    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 16:42:57.690691 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685903    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 16:42:57.690691 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685907    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 16:42:57.690691 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685911    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 16:42:57.690691 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685915    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 16:42:57.690691 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685920    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 16:42:57.690691 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685924    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 16:42:57.690691 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685929    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 16:42:57.690691 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685951    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 16:42:57.690691 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685956    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 16:42:57.690691 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685960    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 16:42:57.690691 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685964    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 16:42:57.690691 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685969    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 16:42:57.691195 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685973    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 16:42:57.691195 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685977    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 16:42:57.691195 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685981    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 16:42:57.691195 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685986    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 16:42:57.691195 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685992    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 16:42:57.691195 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.685997    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 16:42:57.691195 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686001    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 16:42:57.691195 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686006    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 16:42:57.691195 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686010    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 16:42:57.691195 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686014    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 16:42:57.691195 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686018    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 16:42:57.691195 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686022    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 16:42:57.691195 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686026    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 16:42:57.691195 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686030    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 16:42:57.691195 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686034    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 16:42:57.691195 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686039    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 16:42:57.691195 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686043    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 16:42:57.691195 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686047    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 16:42:57.691195 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686051    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 16:42:57.691761 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686055    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 16:42:57.691761 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686060    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 16:42:57.691761 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686064    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 16:42:57.691761 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686069    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 16:42:57.691761 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686074    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 16:42:57.691761 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686079    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 16:42:57.691761 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686083    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 16:42:57.691761 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686089    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 16:42:57.691761 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686100    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 16:42:57.691761 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686104    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 16:42:57.691761 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686108    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 16:42:57.691761 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686112    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 16:42:57.691761 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686116    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 16:42:57.691761 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686120    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 16:42:57.691761 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686128    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 16:42:57.691761 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686134    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 16:42:57.691761 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686139    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 16:42:57.691761 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686146    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 16:42:57.691761 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686150    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 16:42:57.691761 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686154    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 16:42:57.692394 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686159    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 16:42:57.692394 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686164    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 16:42:57.692394 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686169    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 16:42:57.692394 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686173    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 16:42:57.692394 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686177    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 16:42:57.692394 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686182    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 16:42:57.692394 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686186    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 16:42:57.692394 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686191    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 16:42:57.692394 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686195    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 16:42:57.692394 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686200    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 16:42:57.692394 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686205    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 16:42:57.692394 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686210    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 16:42:57.692394 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686215    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 16:42:57.692394 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686219    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 16:42:57.692394 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686225    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 16:42:57.692394 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686229    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 16:42:57.692394 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686236    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 16:42:57.692394 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686240    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 16:42:57.692394 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686244    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 16:42:57.692394 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686249    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 16:42:57.692910 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686253    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 16:42:57.692910 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686257    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 16:42:57.692910 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686261    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 16:42:57.692910 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686266    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 16:42:57.692910 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686271    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 16:42:57.692910 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686275    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 16:42:57.692910 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686280    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 16:42:57.692910 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686283    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 16:42:57.692910 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686288    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 16:42:57.692910 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686294    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 16:42:57.692910 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686299    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 16:42:57.692910 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686303    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 16:42:57.692910 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686307    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 16:42:57.692910 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686311    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 16:42:57.692910 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686317    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 16:42:57.692910 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686323    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 16:42:57.692910 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686329    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 16:42:57.692910 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.686334    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 16:42:57.692910 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686441    2563 flags.go:64] FLAG: --address="0.0.0.0"
Jul 24 16:42:57.692910 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686452    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 24 16:42:57.693416 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686462    2563 flags.go:64] FLAG: --anonymous-auth="true"
Jul 24 16:42:57.693416 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686469    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 24 16:42:57.693416 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686477    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 24 16:42:57.693416 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686482    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 24 16:42:57.693416 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686489    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 24 16:42:57.693416 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686495    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 24 16:42:57.693416 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686500    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 24 16:42:57.693416 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686505    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 24 16:42:57.693416 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686512    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 24 16:42:57.693416 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686517    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 24 16:42:57.693416 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686523    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 24 16:42:57.693416 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686527    2563 flags.go:64] FLAG: --cgroup-root=""
Jul 24 16:42:57.693416 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686533    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 24 16:42:57.693416 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686537    2563 flags.go:64] FLAG: --client-ca-file=""
Jul 24 16:42:57.693416 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686542    2563 flags.go:64] FLAG: --cloud-provider="external"
Jul 24 16:42:57.693416 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686546    2563 flags.go:64] FLAG: --cluster-dns="[]"
Jul 24 16:42:57.693416 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686552    2563 flags.go:64] FLAG: --cluster-domain=""
Jul 24 16:42:57.693416 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686557    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 24 16:42:57.693416 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686562    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 24 16:42:57.693416 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686567    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 24 16:42:57.693416 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686572    2563 flags.go:64] FLAG: --container-log-max-files="5"
Jul 24 16:42:57.693416 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686582    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 24 16:42:57.693416 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686587    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 24 16:42:57.693980 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686592    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 24 16:42:57.693980 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686597    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 24 16:42:57.693980 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686602    2563 flags.go:64] FLAG: --contention-profiling="false"
Jul 24 16:42:57.693980 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686607    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 24 16:42:57.693980 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686612    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 24 16:42:57.693980 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686617    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 24 16:42:57.693980 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686622    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 24 16:42:57.693980 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686628    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 24 16:42:57.693980 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686633    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 24 16:42:57.693980 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686638    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 24 16:42:57.693980 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686642    2563 flags.go:64] FLAG: --enable-load-reader="false"
Jul 24 16:42:57.693980 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686647    2563 flags.go:64] FLAG: --enable-server="true"
Jul 24 16:42:57.693980 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686652    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 24 16:42:57.693980 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686660    2563 flags.go:64] FLAG: --event-burst="100"
Jul 24 16:42:57.693980 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686665    2563 flags.go:64] FLAG: --event-qps="50"
Jul 24 16:42:57.693980 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686669    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 24 16:42:57.693980 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686674    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 24 16:42:57.693980 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686680    2563 flags.go:64] FLAG: --eviction-hard=""
Jul 24 16:42:57.693980 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686687    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 24 16:42:57.693980 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686691    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 24 16:42:57.693980 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686696    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 24 16:42:57.693980 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686701    2563 flags.go:64] FLAG: --eviction-soft=""
Jul 24 16:42:57.693980 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686706    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 24 16:42:57.693980 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686711    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 24 16:42:57.693980 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686715    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 24 16:42:57.694616 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686720    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 24 16:42:57.694616 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686725    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 24 16:42:57.694616 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686729    2563 flags.go:64] FLAG: --fail-swap-on="true"
Jul 24 16:42:57.694616 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686734    2563 flags.go:64] FLAG: --feature-gates=""
Jul 24 16:42:57.694616 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686740    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 24 16:42:57.694616 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686747    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 24 16:42:57.694616 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686752    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 24 16:42:57.694616 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686757    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 24 16:42:57.694616 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686762    2563 flags.go:64] FLAG: --healthz-port="10248"
Jul 24 16:42:57.694616 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686767    2563 flags.go:64] FLAG: --help="false"
Jul 24 16:42:57.694616 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686771    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-133-234.ec2.internal"
Jul 24 16:42:57.694616 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686777    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 24 16:42:57.694616 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686782    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 24 16:42:57.694616 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686786    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 24 16:42:57.694616 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686792    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 24 16:42:57.694616 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686797    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 24 16:42:57.694616 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686802    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 24 16:42:57.694616 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686807    2563 flags.go:64] FLAG: --image-service-endpoint=""
Jul 24 16:42:57.694616 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686812    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 24 16:42:57.694616 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686817    2563 flags.go:64] FLAG: --kube-api-burst="100"
Jul 24 16:42:57.694616 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686823    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 24 16:42:57.694616 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686829    2563 flags.go:64] FLAG: --kube-api-qps="50"
Jul 24 16:42:57.694616 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686834    2563 flags.go:64] FLAG: --kube-reserved=""
Jul 24 16:42:57.694616 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686838    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 24 16:42:57.695323 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686847    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 24 16:42:57.695323 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686852    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 24 16:42:57.695323 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686857    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 24 16:42:57.695323 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686862    2563 flags.go:64] FLAG: --lock-file=""
Jul 24 16:42:57.695323 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686866    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 24 16:42:57.695323 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686871    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 24 16:42:57.695323 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686876    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 24 16:42:57.695323 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686884    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 24 16:42:57.695323 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686889    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 24 16:42:57.695323 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686893    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 24 16:42:57.695323 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686898    2563 flags.go:64] FLAG: --logging-format="text"
Jul 24 16:42:57.695323 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686902    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 24 16:42:57.695323 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686907    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 24 16:42:57.695323 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686913    2563 flags.go:64] FLAG: --manifest-url=""
Jul 24 16:42:57.695323 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686917    2563 flags.go:64] FLAG: --manifest-url-header=""
Jul 24 16:42:57.695323 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686924    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 24 16:42:57.695323 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686929    2563 flags.go:64] FLAG: --max-open-files="1000000"
Jul 24 16:42:57.695323 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686953    2563 flags.go:64] FLAG: --max-pods="110"
Jul 24 16:42:57.695323 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686958    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 24 16:42:57.695323 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686963    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 24 16:42:57.695323 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686967    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 24 16:42:57.695323 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686972    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 24 16:42:57.695323 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686977    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 24 16:42:57.695323 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686982    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 24 16:42:57.695323 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686987    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 24 16:42:57.696002 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.686998    2563 flags.go:64] FLAG: --node-status-max-images="50"
Jul 24 16:42:57.696002 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687002    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 24 16:42:57.696002 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687007    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 24 16:42:57.696002 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687012    2563 flags.go:64] FLAG: --pod-cidr=""
Jul 24 16:42:57.696002 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687018    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 24 16:42:57.696002 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687025    2563 flags.go:64] FLAG: --pod-manifest-path=""
Jul 24 16:42:57.696002 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687030    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 24 16:42:57.696002 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687037    2563 flags.go:64] FLAG: --pods-per-core="0"
Jul 24 16:42:57.696002 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687041    2563 flags.go:64] FLAG: --port="10250"
Jul 24 16:42:57.696002 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687046    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 24 16:42:57.696002 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687051    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e8fbe3621d429eaf"
Jul 24 16:42:57.696002 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687056    2563 flags.go:64] FLAG: --qos-reserved=""
Jul 24 16:42:57.696002 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687061    2563 flags.go:64] FLAG: --read-only-port="10255"
Jul 24 16:42:57.696002 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687065    2563 flags.go:64] FLAG: --register-node="true"
Jul 24 16:42:57.696002 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687070    2563 flags.go:64] FLAG: --register-with-taints=""
Jul 24 16:42:57.696002 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687076    2563 flags.go:64] FLAG: --registry-burst="10"
Jul 24 16:42:57.696002 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687080    2563 flags.go:64] FLAG: --registry-qps="5"
Jul 24 16:42:57.696002 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687085    2563 flags.go:64] FLAG: --reserved-cpus=""
Jul 24 16:42:57.696002 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687090    2563 flags.go:64] FLAG: --reserved-memory=""
Jul 24 16:42:57.696002 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687096    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 24 16:42:57.696002 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687100    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 24 16:42:57.696002 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687106    2563 flags.go:64] FLAG: --rotate-certificates="false"
Jul 24 16:42:57.696002 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687110    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 24 16:42:57.696002 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687114    2563 flags.go:64] FLAG: --runonce="false"
Jul 24 16:42:57.696002 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687119    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 24 16:42:57.696633 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687124    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 24 16:42:57.696633 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687129    2563 flags.go:64] FLAG: --seccomp-default="false"
Jul 24 16:42:57.696633 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687134    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 24 16:42:57.696633 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687138    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 24 16:42:57.696633 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687143    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 24 16:42:57.696633 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687148    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 24 16:42:57.696633 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687152    2563 flags.go:64] FLAG: --storage-driver-password="root"
Jul 24 16:42:57.696633 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687157    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 24 16:42:57.696633 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687161    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 24 16:42:57.696633 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687166    2563 flags.go:64] FLAG: --storage-driver-user="root"
Jul 24 16:42:57.696633 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687171    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 24 16:42:57.696633 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687176    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 24 16:42:57.696633 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687181    2563 flags.go:64] FLAG: --system-cgroups=""
Jul 24 16:42:57.696633 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687186    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 24 16:42:57.696633 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687196    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 24 16:42:57.696633 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687201    2563 flags.go:64] FLAG: --tls-cert-file=""
Jul 24 16:42:57.696633 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687205    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 24 16:42:57.696633 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687212    2563 flags.go:64] FLAG: --tls-min-version=""
Jul 24 16:42:57.696633 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687216    2563 flags.go:64] FLAG: --tls-private-key-file=""
Jul 24 16:42:57.696633 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687220    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 24 16:42:57.696633 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687225    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 24 16:42:57.696633 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687230    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 24 16:42:57.696633 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687235    2563 flags.go:64] FLAG: --v="2"
Jul 24 16:42:57.696633 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687241    2563 flags.go:64] FLAG: --version="false"
Jul 24 16:42:57.696633 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687247    2563 flags.go:64] FLAG: --vmodule=""
Jul 24 16:42:57.697293 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687253    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 24 16:42:57.697293 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.687258    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 24 16:42:57.697293 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687436    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 16:42:57.697293 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687442    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 16:42:57.697293 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687447    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 16:42:57.697293 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687451    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 16:42:57.697293 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687456    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 16:42:57.697293 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687460    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 16:42:57.697293 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687465    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 16:42:57.697293 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687469    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 16:42:57.697293 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687473    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 16:42:57.697293 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687478    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 16:42:57.697293 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687482    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 16:42:57.697293 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687486    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 16:42:57.697293 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687490    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 16:42:57.697293 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687494    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 16:42:57.697293 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687499    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 16:42:57.697293 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687504    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 16:42:57.697293 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687508    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 16:42:57.697293 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687512    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 16:42:57.697776 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687517    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 16:42:57.697776 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687523    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 16:42:57.697776 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687528    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 16:42:57.697776 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687532    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 16:42:57.697776 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687536    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 16:42:57.697776 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687541    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 16:42:57.697776 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687545    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 16:42:57.697776 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687550    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 16:42:57.697776 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687554    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 16:42:57.697776 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687558    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 16:42:57.697776 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687562    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 16:42:57.697776 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687566    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 16:42:57.697776 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687571    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 16:42:57.697776 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687575    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 16:42:57.697776 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687579    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 16:42:57.697776 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687583    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 16:42:57.697776 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687588    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 16:42:57.697776 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687592    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 16:42:57.697776 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687596    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 16:42:57.697776 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687601    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 16:42:57.698286 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687605    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 16:42:57.698286 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687609    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 16:42:57.698286 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687613    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 16:42:57.698286 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687620    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 16:42:57.698286 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687625    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 16:42:57.698286 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687630    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 16:42:57.698286 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687635    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 16:42:57.698286 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687640    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 16:42:57.698286 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687644    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 16:42:57.698286 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687650    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 16:42:57.698286 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687656    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 16:42:57.698286 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687660    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 16:42:57.698286 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687664    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 16:42:57.698286 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687672    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 16:42:57.698286 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687677    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 16:42:57.698286 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687681    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 16:42:57.698286 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687685    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 16:42:57.698286 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687690    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 16:42:57.698286 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687694    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 16:42:57.698746 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687698    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 16:42:57.698746 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687702    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 16:42:57.698746 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687709    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 16:42:57.698746 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687715    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 16:42:57.698746 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687719    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 16:42:57.698746 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687724    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 16:42:57.698746 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687729    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 16:42:57.698746 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687733    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 16:42:57.698746 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687737    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 16:42:57.698746 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687741    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 16:42:57.698746 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687745    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 16:42:57.698746 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687750    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 16:42:57.698746 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687754    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 16:42:57.698746 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687758    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 16:42:57.698746 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687762    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 16:42:57.698746 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687766    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 16:42:57.698746 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687771    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 16:42:57.698746 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687775    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 16:42:57.698746 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687779    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 16:42:57.699243 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687784    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 16:42:57.699243 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687788    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 16:42:57.699243 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687792    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 16:42:57.699243 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687796    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 16:42:57.699243 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687801    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 16:42:57.699243 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687805    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 16:42:57.699243 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687809    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 16:42:57.699243 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687814    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 16:42:57.699243 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687824    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 16:42:57.699243 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687829    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 16:42:57.699243 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687834    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 16:42:57.699243 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687838    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 16:42:57.699243 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687842    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 16:42:57.699243 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687847    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 16:42:57.699243 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687851    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 16:42:57.699243 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687855    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 16:42:57.699243 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687860    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 16:42:57.699243 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687864    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 16:42:57.699243 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687868    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 16:42:57.699243 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687872    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 16:42:57.699735 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687877    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 16:42:57.699735 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687881    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 16:42:57.699735 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.687885    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 16:42:57.699735 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.688862    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 16:42:57.699735 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.696764    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 24 16:42:57.699735 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.696779    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 24 16:42:57.699735 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.697645    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 24 16:42:57.699735 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.697655    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 24 16:42:57.699735 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697719    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 16:42:57.699735 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697724    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 16:42:57.699735 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697728    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 16:42:57.699735 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697731    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 16:42:57.699735 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697734    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 16:42:57.699735 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697736    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 16:42:57.700112 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697739    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 16:42:57.700112 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697742    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 16:42:57.700112 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697745    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 16:42:57.700112 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697749    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 16:42:57.700112 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697753    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 16:42:57.700112 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697756    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 16:42:57.700112 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697765    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 16:42:57.700112 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697768    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 16:42:57.700112 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697771    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 16:42:57.700112 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697774    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 16:42:57.700112 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697782    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 16:42:57.700112 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697785    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 16:42:57.700112 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697788    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 16:42:57.700112 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697791    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 16:42:57.700112 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697793    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 16:42:57.700112 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697796    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 16:42:57.700112 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697798    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 16:42:57.700112 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697801    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 16:42:57.700112 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697803    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 16:42:57.700559 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697806    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 16:42:57.700559 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697808    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 16:42:57.700559 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697811    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 16:42:57.700559 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697814    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 16:42:57.700559 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697818    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 16:42:57.700559 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697820    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 16:42:57.700559 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697823    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 16:42:57.700559 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697826    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 16:42:57.700559 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697829    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 16:42:57.700559 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697831    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 16:42:57.700559 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697839    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 16:42:57.700559 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697842    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 16:42:57.700559 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697844    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 16:42:57.700559 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697847    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 16:42:57.700559 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697849    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 16:42:57.700559 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697852    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 16:42:57.700559 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697854    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 16:42:57.700559 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697857    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 16:42:57.700559 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697860    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 16:42:57.700559 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697862    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 16:42:57.701117 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697870    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 16:42:57.701117 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697873    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 16:42:57.701117 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697876    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 16:42:57.701117 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697878    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 16:42:57.701117 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697881    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 16:42:57.701117 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697883    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 16:42:57.701117 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697886    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 16:42:57.701117 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697888    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 16:42:57.701117 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697891    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 16:42:57.701117 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697893    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 16:42:57.701117 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697895    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 16:42:57.701117 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697898    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 16:42:57.701117 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697900    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 16:42:57.701117 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697903    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 16:42:57.701117 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697905    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 16:42:57.701117 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697909    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 16:42:57.701117 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697913    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 16:42:57.701117 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697917    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 16:42:57.701117 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697920    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 16:42:57.701117 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697923    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 16:42:57.701611 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697926    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 16:42:57.701611 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697928    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 16:42:57.701611 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697931    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 16:42:57.701611 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697944    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 16:42:57.701611 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697947    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 16:42:57.701611 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697949    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 16:42:57.701611 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697952    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 16:42:57.701611 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697954    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 16:42:57.701611 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697957    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 16:42:57.701611 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697960    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 16:42:57.701611 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697962    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 16:42:57.701611 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697965    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 16:42:57.701611 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697967    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 16:42:57.701611 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697976    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 16:42:57.701611 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697978    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 16:42:57.701611 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697981    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 16:42:57.701611 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697984    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 16:42:57.701611 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697986    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 16:42:57.701611 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697989    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 16:42:57.701611 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697991    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 16:42:57.702112 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697994    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 16:42:57.702112 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.697996    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 16:42:57.702112 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698000    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 16:42:57.702112 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698003    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 16:42:57.702112 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698006    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 16:42:57.702112 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698009    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 16:42:57.702112 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698012    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 16:42:57.702112 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698015    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 16:42:57.702112 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698018    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 16:42:57.702112 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698021    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 16:42:57.702112 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698023    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 16:42:57.702112 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698026    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 16:42:57.702112 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698028    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 16:42:57.702112 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698031    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 16:42:57.702112 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.698036    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 16:42:57.702479 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698209    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 16:42:57.702479 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698215    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 16:42:57.702479 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698218    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 16:42:57.702479 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698221    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 16:42:57.702479 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698224    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 16:42:57.702479 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698226    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 16:42:57.702479 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698229    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 16:42:57.702479 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698231    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 16:42:57.702479 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698234    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 16:42:57.702479 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698237    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 16:42:57.702479 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698239    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 16:42:57.702479 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698247    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 16:42:57.702479 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698250    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 16:42:57.702479 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698252    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 16:42:57.702479 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698255    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 16:42:57.702479 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698257    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 16:42:57.702479 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698260    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 16:42:57.702479 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698262    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 16:42:57.702479 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698265    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 16:42:57.702946 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698267    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 16:42:57.702946 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698270    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 16:42:57.702946 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698272    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 16:42:57.702946 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698275    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 16:42:57.702946 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698277    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 16:42:57.702946 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698280    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 16:42:57.702946 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698282    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 16:42:57.702946 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698286    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 16:42:57.702946 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698289    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 16:42:57.702946 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698293    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 16:42:57.702946 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698296    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 16:42:57.702946 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698299    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 16:42:57.702946 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698303    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 16:42:57.702946 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698305    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 16:42:57.702946 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698308    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 16:42:57.702946 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698311    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 16:42:57.702946 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698313    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 16:42:57.702946 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698316    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 16:42:57.702946 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698319    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 16:42:57.702946 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698322    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 16:42:57.703446 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698325    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 16:42:57.703446 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698327    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 16:42:57.703446 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698330    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 16:42:57.703446 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698333    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 16:42:57.703446 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698335    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 16:42:57.703446 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698345    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 16:42:57.703446 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698348    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 16:42:57.703446 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698350    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 16:42:57.703446 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698353    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 16:42:57.703446 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698356    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 16:42:57.703446 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698359    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 16:42:57.703446 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698362    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 16:42:57.703446 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698364    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 16:42:57.703446 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698367    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 16:42:57.703446 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698370    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 16:42:57.703446 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698372    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 16:42:57.703446 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698375    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 16:42:57.703446 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698379    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 16:42:57.703446 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698382    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 16:42:57.703909 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698385    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 16:42:57.703909 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698387    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 16:42:57.703909 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698390    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 16:42:57.703909 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698392    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 16:42:57.703909 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698395    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 16:42:57.703909 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698397    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 16:42:57.703909 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698400    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 16:42:57.703909 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698402    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 16:42:57.703909 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698405    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 16:42:57.703909 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698407    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 16:42:57.703909 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698410    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 16:42:57.703909 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698412    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 16:42:57.703909 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698415    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 16:42:57.703909 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698417    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 16:42:57.703909 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698420    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 16:42:57.703909 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698422    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 16:42:57.703909 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698425    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 16:42:57.703909 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698445    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 16:42:57.703909 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698449    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 16:42:57.703909 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698458    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 16:42:57.704412 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698461    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 16:42:57.704412 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698463    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 16:42:57.704412 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698466    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 16:42:57.704412 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698468    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 16:42:57.704412 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698471    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 16:42:57.704412 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698473    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 16:42:57.704412 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698476    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 16:42:57.704412 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698479    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 16:42:57.704412 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698481    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 16:42:57.704412 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698483    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 16:42:57.704412 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698486    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 16:42:57.704412 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698489    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 16:42:57.704412 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698491    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 16:42:57.704412 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698494    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 16:42:57.704412 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698497    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 16:42:57.704412 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698499    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 16:42:57.704412 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698502    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 16:42:57.704412 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698504    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 16:42:57.704412 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698507    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 16:42:57.704412 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698509    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 16:42:57.704910 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:57.698512    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 16:42:57.704910 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.698516    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 16:42:57.704910 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.698645    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 24 16:42:57.704910 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.701170    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 24 16:42:57.704910 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.702811    2563 server.go:1019] "Starting client certificate rotation"
Jul 24 16:42:57.704910 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.702914    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 16:42:57.704910 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.702960    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 16:42:57.705660 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.705645    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 16:42:57.709321 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.709300    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 16:42:57.719712 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.719690    2563 log.go:25] "Validated CRI v1 runtime API"
Jul 24 16:42:57.722105 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.722079    2563 log.go:25] "Validated CRI v1 image API"
Jul 24 16:42:57.724689 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.724675    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 24 16:42:57.727191 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.727165    2563 fs.go:135] Filesystem UUIDs: map[00c7c3cc-393a-4d0d-b428-ec2252b65633:/dev/nvme0n1p3 5bda3ccf-c685-4d39-88ab-45ba095bf507:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jul 24 16:42:57.727276 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.727210    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 24 16:42:57.732311 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.732201    2563 manager.go:216] Machine: {Timestamp:2026-07-24 16:42:57.730893628 +0000 UTC m=+0.372857514 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3106027 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec246283dd218e3879de95668160edb8 SystemUUID:ec246283-dd21-8e38-79de-95668160edb8 BootID:e4368b76-22bb-4ccb-83f3-d460163de417 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:be:2b:da:9e:75 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:be:2b:da:9e:75 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:82:42:b3:a8:3c:5d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 24 16:42:57.732311 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.732298    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 24 16:42:57.732473 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.732391    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 24 16:42:57.732868 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.732852    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 16:42:57.733678 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.733652    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 24 16:42:57.733804 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.733681    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-234.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 24 16:42:57.734541 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.734528    2563 topology_manager.go:138] "Creating topology manager with none policy"
Jul 24 16:42:57.734586 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.734547    2563 container_manager_linux.go:309] "Creating device plugin manager"
Jul 24 16:42:57.734586 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.734558    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 16:42:57.735467 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.735456    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 16:42:57.736666 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.736656    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 24 16:42:57.739519 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.739508    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 16:42:57.739628 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.739618    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 24 16:42:57.739660 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.739648    2563 kubelet.go:486] "Attempting to sync node with API server"
Jul 24 16:42:57.739660 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.739657    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 24 16:42:57.739714 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.739669    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 24 16:42:57.739714 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.739680    2563 kubelet.go:398] "Adding apiserver pod source"
Jul 24 16:42:57.739714 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.739688    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 24 16:42:57.740113 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.740090    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 16:42:57.740194 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.740120    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 16:42:57.748840 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.748820    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 24 16:42:57.751221 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.751207    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 24 16:42:57.751284 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.751236    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 24 16:42:57.753150 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.753137    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 24 16:42:57.753202 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.753154    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 24 16:42:57.753202 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.753160    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 24 16:42:57.753202 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.753165    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 24 16:42:57.753202 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.753174    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 24 16:42:57.753202 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.753180    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 24 16:42:57.753202 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.753188    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 24 16:42:57.753202 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.753193    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 24 16:42:57.753202 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.753199    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 24 16:42:57.753202 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.753205    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 24 16:42:57.753543 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.753213    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 24 16:42:57.753543 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.753218    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 24 16:42:57.753929 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.753917    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 24 16:42:57.753929 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.753928    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 24 16:42:57.754248 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:57.754227    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-234.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 24 16:42:57.754297 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:57.754230    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 24 16:42:57.756880 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.756864    2563 server.go:1268] "Started kubelet"
Jul 24 16:42:57.757002 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.756917    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 24 16:42:57.757067 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.756993    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 24 16:42:57.757067 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.757051    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 24 16:42:57.757646 ip-10-0-133-234 systemd[1]: Started Kubernetes Kubelet.
Jul 24 16:42:57.759839 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.759823    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 24 16:42:57.761380 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.760876    2563 server.go:310] "Adding debug handlers to kubelet server"
Jul 24 16:42:57.763243 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:57.763223    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 24 16:42:57.766382 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.766368    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-234.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 24 16:42:57.767441 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:57.766405    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-234.ec2.internal.18c54770f2bfd1b9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-234.ec2.internal,UID:ip-10-0-133-234.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-234.ec2.internal,},FirstTimestamp:2026-07-24 16:42:57.756836281 +0000 UTC m=+0.398800156,LastTimestamp:2026-07-24 16:42:57.756836281 +0000 UTC m=+0.398800156,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-234.ec2.internal,}"
Jul 24 16:42:57.768504 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.768486    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 24 16:42:57.768504 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.768496    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 24 16:42:57.768634 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:57.768600    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-234.ec2.internal\" not found"
Jul 24 16:42:57.769231 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.769213    2563 factory.go:55] Registering systemd factory
Jul 24 16:42:57.769231 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.769231    2563 factory.go:223] Registration of the systemd container factory successfully
Jul 24 16:42:57.769447 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.769273    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 24 16:42:57.769447 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.769276    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 24 16:42:57.769447 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.769283    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 24 16:42:57.769447 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.769384    2563 reconstruct.go:97] "Volume reconstruction finished"
Jul 24 16:42:57.769447 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.769394    2563 reconciler.go:29] "Reconciler: start to sync state"
Jul 24 16:42:57.769447 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.769430    2563 factory.go:153] Registering CRI-O factory
Jul 24 16:42:57.769447 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.769444    2563 factory.go:223] Registration of the crio container factory successfully
Jul 24 16:42:57.769694 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.769491    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 24 16:42:57.769694 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.769512    2563 factory.go:103] Registering Raw factory
Jul 24 16:42:57.769694 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.769536    2563 manager.go:1196] Started watching for new ooms in manager
Jul 24 16:42:57.769947 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.769919    2563 manager.go:349] Starting recovery of all containers
Jul 24 16:42:57.770158 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:57.770135    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-234.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 24 16:42:57.770328 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:57.770297    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 24 16:42:57.777627 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.777605    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fmlww"
Jul 24 16:42:57.782957 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.782842    2563 manager.go:354] Recovery completed
Jul 24 16:42:57.783022 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.783004    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fmlww"
Jul 24 16:42:57.788160 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.788147    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 16:42:57.789878 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.789864    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 16:42:57.789963 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.789891    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 16:42:57.789963 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.789903    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasSufficientPID"
Jul 24 16:42:57.790496 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.790478    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 24 16:42:57.790580 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.790497    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 16:42:57.790580 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.790497    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 24 16:42:57.790580 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.790550    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 16:42:57.791160 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.791144    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 16:42:57.791221 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.791176    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 16:42:57.791221 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.791190    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasSufficientPID"
Jul 24 16:42:57.793311 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.793298    2563 policy_none.go:49] "None policy: Start"
Jul 24 16:42:57.793375 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.793315    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 24 16:42:57.793375 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.793329    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 24 16:42:57.795289 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.795276    2563 policy_none.go:47] "Start"
Jul 24 16:42:57.800963 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:57.800869    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-234.ec2.internal.18c54770f4b80059  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-234.ec2.internal,UID:ip-10-0-133-234.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-234.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-234.ec2.internal,},FirstTimestamp:2026-07-24 16:42:57.789878361 +0000 UTC m=+0.431842239,LastTimestamp:2026-07-24 16:42:57.789878361 +0000 UTC m=+0.431842239,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-234.ec2.internal,}"
Jul 24 16:42:57.810191 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:57.810110    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-234.ec2.internal.18c54770f4b84d14  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-234.ec2.internal,UID:ip-10-0-133-234.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-234.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-234.ec2.internal,},FirstTimestamp:2026-07-24 16:42:57.789898004 +0000 UTC m=+0.431861879,LastTimestamp:2026-07-24 16:42:57.789898004 +0000 UTC m=+0.431861879,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-234.ec2.internal,}"
Jul 24 16:42:57.820250 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:57.820194    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-234.ec2.internal.18c54770f4b87319  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-234.ec2.internal,UID:ip-10-0-133-234.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-234.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-234.ec2.internal,},FirstTimestamp:2026-07-24 16:42:57.789907737 +0000 UTC m=+0.431871612,LastTimestamp:2026-07-24 16:42:57.789907737 +0000 UTC m=+0.431871612,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-234.ec2.internal,}"
Jul 24 16:42:57.832120 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:57.832059    2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-234.ec2.internal.18c54770f4b80059\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-234.ec2.internal.18c54770f4b80059  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-234.ec2.internal,UID:ip-10-0-133-234.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-234.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-234.ec2.internal,},FirstTimestamp:2026-07-24 16:42:57.789878361 +0000 UTC m=+0.431842239,LastTimestamp:2026-07-24 16:42:57.791159704 +0000 UTC m=+0.433123580,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-234.ec2.internal,}"
Jul 24 16:42:57.835481 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.835467    2563 manager.go:337] "Starting Device Plugin manager"
Jul 24 16:42:57.835544 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:57.835501    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 24 16:42:57.835544 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.835511    2563 server.go:87] "Starting device plugin registration server"
Jul 24 16:42:57.835742 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.835730    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 24 16:42:57.835792 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.835743    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 24 16:42:57.835848 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.835821    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 24 16:42:57.835900 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.835894    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 24 16:42:57.835964 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.835903    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 24 16:42:57.836401 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:57.836383    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 24 16:42:57.836469 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:57.836436    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-234.ec2.internal\" not found"
Jul 24 16:42:57.839961 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:57.839879    2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-234.ec2.internal.18c54770f4b84d14\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-234.ec2.internal.18c54770f4b84d14  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-234.ec2.internal,UID:ip-10-0-133-234.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-234.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-234.ec2.internal,},FirstTimestamp:2026-07-24 16:42:57.789898004 +0000 UTC m=+0.431861879,LastTimestamp:2026-07-24 16:42:57.791183356 +0000 UTC m=+0.433147233,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-234.ec2.internal,}"
Jul 24 16:42:57.849027 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:57.848968    2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-234.ec2.internal.18c54770f4b87319\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-234.ec2.internal.18c54770f4b87319  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-234.ec2.internal,UID:ip-10-0-133-234.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-234.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-234.ec2.internal,},FirstTimestamp:2026-07-24 16:42:57.789907737 +0000 UTC m=+0.431871612,LastTimestamp:2026-07-24 16:42:57.791195314 +0000 UTC m=+0.433159190,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-234.ec2.internal,}"
Jul 24 16:42:57.860371 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:57.860316    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-234.ec2.internal.18c54770f790522d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-234.ec2.internal,UID:ip-10-0-133-234.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-133-234.ec2.internal,},FirstTimestamp:2026-07-24 16:42:57.837609517 +0000 UTC m=+0.479573383,LastTimestamp:2026-07-24 16:42:57.837609517 +0000 UTC m=+0.479573383,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-234.ec2.internal,}"
Jul 24 16:42:57.936695 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.936616    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 16:42:57.937609 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.937473    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 16:42:57.937676 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.937627    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 16:42:57.937676 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.937642    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasSufficientPID"
Jul 24 16:42:57.937676 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.937666    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-234.ec2.internal"
Jul 24 16:42:57.954051 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.954032    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-234.ec2.internal"
Jul 24 16:42:57.967901 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:57.954054    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-234.ec2.internal\": node \"ip-10-0-133-234.ec2.internal\" not found"
Jul 24 16:42:57.978304 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:57.978284    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-234.ec2.internal\" not found"
Jul 24 16:42:57.994631 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.994602    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 24 16:42:57.995757 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.995740    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 24 16:42:57.995839 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.995773    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 24 16:42:57.995839 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.995792    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 24 16:42:57.995839 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:57.995822    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 24 16:42:57.997847 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:57.997830    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 16:42:58.079175 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:58.079142    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-234.ec2.internal\" not found"
Jul 24 16:42:58.096306 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.096283    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-234.ec2.internal"]
Jul 24 16:42:58.096383 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.096346    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 16:42:58.097274 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.097253    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 16:42:58.097357 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.097282    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 16:42:58.097357 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.097294    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasSufficientPID"
Jul 24 16:42:58.099627 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.099615    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 16:42:58.099762 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.099749    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal"
Jul 24 16:42:58.099795 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.099780    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 16:42:58.100374 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.100355    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 16:42:58.100447 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.100397    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 16:42:58.100447 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.100414    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 16:42:58.100447 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.100423    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasSufficientPID"
Jul 24 16:42:58.100545 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.100398    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 16:42:58.100545 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.100487    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasSufficientPID"
Jul 24 16:42:58.102660 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.102647    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-234.ec2.internal"
Jul 24 16:42:58.102713 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.102671    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 16:42:58.103353 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.103340    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 16:42:58.103410 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.103376    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 16:42:58.103410 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.103388    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasSufficientPID"
Jul 24 16:42:58.123472 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:58.123449    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-234.ec2.internal\" not found" node="ip-10-0-133-234.ec2.internal"
Jul 24 16:42:58.127751 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:58.127735    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-234.ec2.internal\" not found" node="ip-10-0-133-234.ec2.internal"
Jul 24 16:42:58.170475 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.170454    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5737b582bc5c83055d2673f557e5384a-config\") pod \"kube-apiserver-proxy-ip-10-0-133-234.ec2.internal\" (UID: \"5737b582bc5c83055d2673f557e5384a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-234.ec2.internal"
Jul 24 16:42:58.170538 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.170478    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5cdfb9b825abe617ad3cdd7a8d779a3b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal\" (UID: \"5cdfb9b825abe617ad3cdd7a8d779a3b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal"
Jul 24 16:42:58.170538 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.170494    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cdfb9b825abe617ad3cdd7a8d779a3b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal\" (UID: \"5cdfb9b825abe617ad3cdd7a8d779a3b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal"
Jul 24 16:42:58.180054 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:58.180037    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-234.ec2.internal\" not found"
Jul 24 16:42:58.270651 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.270596    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5cdfb9b825abe617ad3cdd7a8d779a3b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal\" (UID: \"5cdfb9b825abe617ad3cdd7a8d779a3b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal"
Jul 24 16:42:58.270726 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.270636    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5cdfb9b825abe617ad3cdd7a8d779a3b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal\" (UID: \"5cdfb9b825abe617ad3cdd7a8d779a3b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal"
Jul 24 16:42:58.270726 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.270684    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cdfb9b825abe617ad3cdd7a8d779a3b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal\" (UID: \"5cdfb9b825abe617ad3cdd7a8d779a3b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal"
Jul 24 16:42:58.270726 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.270707    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5737b582bc5c83055d2673f557e5384a-config\") pod \"kube-apiserver-proxy-ip-10-0-133-234.ec2.internal\" (UID: \"5737b582bc5c83055d2673f557e5384a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-234.ec2.internal"
Jul 24 16:42:58.270815 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.270767    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5737b582bc5c83055d2673f557e5384a-config\") pod \"kube-apiserver-proxy-ip-10-0-133-234.ec2.internal\" (UID: \"5737b582bc5c83055d2673f557e5384a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-234.ec2.internal"
Jul 24 16:42:58.270815 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.270774    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cdfb9b825abe617ad3cdd7a8d779a3b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal\" (UID: \"5cdfb9b825abe617ad3cdd7a8d779a3b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal"
Jul 24 16:42:58.280681 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:58.280663    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-234.ec2.internal\" not found"
Jul 24 16:42:58.381162 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:58.381138    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-234.ec2.internal\" not found"
Jul 24 16:42:58.428326 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.428299    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal"
Jul 24 16:42:58.430479 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.430464    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-234.ec2.internal"
Jul 24 16:42:58.481382 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:58.481345    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-234.ec2.internal\" not found"
Jul 24 16:42:58.582444 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:58.582383    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-234.ec2.internal\" not found"
Jul 24 16:42:58.683260 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:58.683209    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-234.ec2.internal\" not found"
Jul 24 16:42:58.702839 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.702809    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 24 16:42:58.702985 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.702964    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 24 16:42:58.718871 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.718851    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 16:42:58.719253 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.719236    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 16:42:58.739844 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.739818    2563 apiserver.go:52] "Watching apiserver"
Jul 24 16:42:58.750616 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.750591    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 24 16:42:58.750974 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.750937    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-vq4cx","openshift-multus/multus-additional-cni-plugins-t4xnt","openshift-multus/multus-dblpg","openshift-network-diagnostics/network-check-target-hhvqg","kube-system/global-pull-secret-syncer-vzbnm","kube-system/konnectivity-agent-45v2q","openshift-multus/network-metrics-daemon-8lwz8","openshift-network-operator/iptables-alerter-gt8zs","openshift-ovn-kubernetes/ovnkube-node-gt5dk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfw5x","openshift-cluster-node-tuning-operator/tuned-mvlt2","openshift-dns/node-resolver-g5sl6"]
Jul 24 16:42:58.753680 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.753662    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vq4cx"
Jul 24 16:42:58.755945 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.755926    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t4xnt"
Jul 24 16:42:58.756405 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.756387    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dp5dv\""
Jul 24 16:42:58.756886 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.756863    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 24 16:42:58.757036 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.756889    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 24 16:42:58.757036 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.756867    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 24 16:42:58.758295 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.758272    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.758398 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.758381    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 24 16:42:58.758741 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.758727    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 24 16:42:58.758803 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.758758    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 24 16:42:58.759053 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.759030    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 24 16:42:58.759133 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.759067    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7nlsp\""
Jul 24 16:42:58.759251 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.759234    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 24 16:42:58.760536 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.760515    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hhvqg"
Jul 24 16:42:58.760610 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.760566    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 24 16:42:58.760651 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:58.760616    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hhvqg" podUID="a12b44d3-5fd7-4543-bf4b-6b9ba880c11b"
Jul 24 16:42:58.760874 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.760860    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-w9gzx\""
Jul 24 16:42:58.762809 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.762794    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vzbnm"
Jul 24 16:42:58.762879 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:58.762861    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vzbnm" podUID="573b3eaf-dd83-456c-b7e0-35cb5aad6884"
Jul 24 16:42:58.765041 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.765023    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-45v2q"
Jul 24 16:42:58.767391 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.767369    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lwz8"
Jul 24 16:42:58.767512 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:58.767492    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lwz8" podUID="54f4b6d0-be12-40ae-b3ac-f7acb8e715b9"
Jul 24 16:42:58.767596 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.767515    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2s7z6\""
Jul 24 16:42:58.767596 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.767583    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 24 16:42:58.767705 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.767618    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 24 16:42:58.768611 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.768592    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-234.ec2.internal"
Jul 24 16:42:58.768719 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.768641    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 24 16:42:58.769903 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.769879    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gt8zs"
Jul 24 16:42:58.771994 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.771932    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wcpjh\""
Jul 24 16:42:58.772091 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.772073    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 24 16:42:58.772183 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.772168    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 24 16:42:58.772247 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.772214    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 24 16:42:58.772494 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.772479    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.774936 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.774917    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 24 16:42:58.775188 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.775169    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfw5x"
Jul 24 16:42:58.775377 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.775352    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 24 16:42:58.775489 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.775447    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 24 16:42:58.775561 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.775505    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 24 16:42:58.775753 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.775737    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cxqsg\""
Jul 24 16:42:58.776080 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.776063    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 24 16:42:58.776162 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.776081    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 24 16:42:58.776947 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.776928    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzm9x\" (UniqueName: \"kubernetes.io/projected/aa45a2eb-0246-4044-85cb-f09d0750d858-kube-api-access-bzm9x\") pod \"iptables-alerter-gt8zs\" (UID: \"aa45a2eb-0246-4044-85cb-f09d0750d858\") " pod="openshift-network-operator/iptables-alerter-gt8zs"
Jul 24 16:42:58.776999 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.776956    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-system-cni-dir\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.776999 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.776972    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-multus-conf-dir\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.777082 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.777018    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/573b3eaf-dd83-456c-b7e0-35cb5aad6884-dbus\") pod \"global-pull-secret-syncer-vzbnm\" (UID: \"573b3eaf-dd83-456c-b7e0-35cb5aad6884\") " pod="kube-system/global-pull-secret-syncer-vzbnm"
Jul 24 16:42:58.777082 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.777044    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54f4b6d0-be12-40ae-b3ac-f7acb8e715b9-metrics-certs\") pod \"network-metrics-daemon-8lwz8\" (UID: \"54f4b6d0-be12-40ae-b3ac-f7acb8e715b9\") " pod="openshift-multus/network-metrics-daemon-8lwz8"
Jul 24 16:42:58.777082 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.777063    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ce9a1972-6a1e-4e0d-9dab-0e25820dedf1-serviceca\") pod \"node-ca-vq4cx\" (UID: \"ce9a1972-6a1e-4e0d-9dab-0e25820dedf1\") " pod="openshift-image-registry/node-ca-vq4cx"
Jul 24 16:42:58.777082 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.777078    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6a0daaab-0ecb-4e33-bc7c-c53caef203f2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t4xnt\" (UID: \"6a0daaab-0ecb-4e33-bc7c-c53caef203f2\") " pod="openshift-multus/multus-additional-cni-plugins-t4xnt"
Jul 24 16:42:58.777256 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.777097    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6597s\" (UniqueName: \"kubernetes.io/projected/6a0daaab-0ecb-4e33-bc7c-c53caef203f2-kube-api-access-6597s\") pod \"multus-additional-cni-plugins-t4xnt\" (UID: \"6a0daaab-0ecb-4e33-bc7c-c53caef203f2\") " pod="openshift-multus/multus-additional-cni-plugins-t4xnt"
Jul 24 16:42:58.777256 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.777126    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6a0daaab-0ecb-4e33-bc7c-c53caef203f2-os-release\") pod \"multus-additional-cni-plugins-t4xnt\" (UID: \"6a0daaab-0ecb-4e33-bc7c-c53caef203f2\") " pod="openshift-multus/multus-additional-cni-plugins-t4xnt"
Jul 24 16:42:58.777256 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.777141    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/855016a9-43a6-4765-815e-9739917af352-konnectivity-ca\") pod \"konnectivity-agent-45v2q\" (UID: \"855016a9-43a6-4765-815e-9739917af352\") " pod="kube-system/konnectivity-agent-45v2q"
Jul 24 16:42:58.777256 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.777157    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-host-var-lib-cni-multus\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.777256 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.777171    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-host-var-lib-kubelet\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.777256 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.777187    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/573b3eaf-dd83-456c-b7e0-35cb5aad6884-kubelet-config\") pod \"global-pull-secret-syncer-vzbnm\" (UID: \"573b3eaf-dd83-456c-b7e0-35cb5aad6884\") " pod="kube-system/global-pull-secret-syncer-vzbnm"
Jul 24 16:42:58.777256 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.777206    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/573b3eaf-dd83-456c-b7e0-35cb5aad6884-original-pull-secret\") pod \"global-pull-secret-syncer-vzbnm\" (UID: \"573b3eaf-dd83-456c-b7e0-35cb5aad6884\") " pod="kube-system/global-pull-secret-syncer-vzbnm"
Jul 24 16:42:58.777256 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.777238    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a0daaab-0ecb-4e33-bc7c-c53caef203f2-cni-binary-copy\") pod \"multus-additional-cni-plugins-t4xnt\" (UID: \"6a0daaab-0ecb-4e33-bc7c-c53caef203f2\") " pod="openshift-multus/multus-additional-cni-plugins-t4xnt"
Jul 24 16:42:58.777641 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.777262    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-host-run-k8s-cni-cncf-io\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.777641 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.777288    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-host-run-netns\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.777641 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.777327    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-hostroot\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.777641 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.777383    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e620ec4e-9703-45c1-9080-d17039c0d2e8-multus-daemon-config\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.777641 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.777426    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmzk2\" (UniqueName: \"kubernetes.io/projected/e620ec4e-9703-45c1-9080-d17039c0d2e8-kube-api-access-qmzk2\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.777641 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.777454    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce9a1972-6a1e-4e0d-9dab-0e25820dedf1-host\") pod \"node-ca-vq4cx\" (UID: \"ce9a1972-6a1e-4e0d-9dab-0e25820dedf1\") " pod="openshift-image-registry/node-ca-vq4cx"
Jul 24 16:42:58.777641 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.777497    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a0daaab-0ecb-4e33-bc7c-c53caef203f2-system-cni-dir\") pod \"multus-additional-cni-plugins-t4xnt\" (UID: \"6a0daaab-0ecb-4e33-bc7c-c53caef203f2\") " pod="openshift-multus/multus-additional-cni-plugins-t4xnt"
Jul 24 16:42:58.777641 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.777529    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aa45a2eb-0246-4044-85cb-f09d0750d858-host-slash\") pod \"iptables-alerter-gt8zs\" (UID: \"aa45a2eb-0246-4044-85cb-f09d0750d858\") " pod="openshift-network-operator/iptables-alerter-gt8zs"
Jul 24 16:42:58.777641 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.777553    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e620ec4e-9703-45c1-9080-d17039c0d2e8-cni-binary-copy\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.777641 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.777579    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a0daaab-0ecb-4e33-bc7c-c53caef203f2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t4xnt\" (UID: \"6a0daaab-0ecb-4e33-bc7c-c53caef203f2\") " pod="openshift-multus/multus-additional-cni-plugins-t4xnt"
Jul 24 16:42:58.777641 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.777602    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/855016a9-43a6-4765-815e-9739917af352-agent-certs\") pod \"konnectivity-agent-45v2q\" (UID: \"855016a9-43a6-4765-815e-9739917af352\") " pod="kube-system/konnectivity-agent-45v2q"
Jul 24 16:42:58.777641 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.777628    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-os-release\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.778168 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.777661    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-multus-socket-dir-parent\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.778168 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.777683    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jshsg\""
Jul 24 16:42:58.778168 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.777688    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-etc-kubernetes\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.778168 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.777732    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a0daaab-0ecb-4e33-bc7c-c53caef203f2-cnibin\") pod \"multus-additional-cni-plugins-t4xnt\" (UID: \"6a0daaab-0ecb-4e33-bc7c-c53caef203f2\") " pod="openshift-multus/multus-additional-cni-plugins-t4xnt"
Jul 24 16:42:58.778168 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.777750    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/aa45a2eb-0246-4044-85cb-f09d0750d858-iptables-alerter-script\") pod \"iptables-alerter-gt8zs\" (UID: \"aa45a2eb-0246-4044-85cb-f09d0750d858\") " pod="openshift-network-operator/iptables-alerter-gt8zs"
Jul 24 16:42:58.778168 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.777774    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-cnibin\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.778168 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.777790    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-host-run-multus-certs\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.778168 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.777815    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndwn5\" (UniqueName: \"kubernetes.io/projected/a12b44d3-5fd7-4543-bf4b-6b9ba880c11b-kube-api-access-ndwn5\") pod \"network-check-target-hhvqg\" (UID: \"a12b44d3-5fd7-4543-bf4b-6b9ba880c11b\") " pod="openshift-network-diagnostics/network-check-target-hhvqg"
Jul 24 16:42:58.778168 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.777839    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2m6p\" (UniqueName: \"kubernetes.io/projected/54f4b6d0-be12-40ae-b3ac-f7acb8e715b9-kube-api-access-r2m6p\") pod \"network-metrics-daemon-8lwz8\" (UID: \"54f4b6d0-be12-40ae-b3ac-f7acb8e715b9\") " pod="openshift-multus/network-metrics-daemon-8lwz8"
Jul 24 16:42:58.778168 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.777859    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.778168 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.777873    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6a0daaab-0ecb-4e33-bc7c-c53caef203f2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t4xnt\" (UID: \"6a0daaab-0ecb-4e33-bc7c-c53caef203f2\") " pod="openshift-multus/multus-additional-cni-plugins-t4xnt"
Jul 24 16:42:58.778168 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.777897    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-multus-cni-dir\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.778168 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.777913    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-host-var-lib-cni-bin\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.778168 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.777968    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kst4t\" (UniqueName: \"kubernetes.io/projected/ce9a1972-6a1e-4e0d-9dab-0e25820dedf1-kube-api-access-kst4t\") pod \"node-ca-vq4cx\" (UID: \"ce9a1972-6a1e-4e0d-9dab-0e25820dedf1\") " pod="openshift-image-registry/node-ca-vq4cx"
Jul 24 16:42:58.778913 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.778264    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 24 16:42:58.778913 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.778269    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 24 16:42:58.778913 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.778364    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 24 16:42:58.778913 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.778510    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 16:42:58.778913 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.778688    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal"
Jul 24 16:42:58.781212 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.780926    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-234.ec2.internal"]
Jul 24 16:42:58.781212 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.781000    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7sk2c\""
Jul 24 16:42:58.781212 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.781000    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 24 16:42:58.781212 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.781088    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g5sl6"
Jul 24 16:42:58.781853 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.781205    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 24 16:42:58.781853 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.781311    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 16:42:58.783987 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.783962    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-v4j7c\""
Jul 24 16:42:58.784116 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.784011    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 24 16:42:58.784386 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.784011    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 24 16:42:58.794113 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.794097    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 16:42:58.794190 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.794153    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal"]
Jul 24 16:42:58.814393 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.814371    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xc459"
Jul 24 16:42:58.821309 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.821293    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xc459"
Jul 24 16:42:58.870901 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.870874    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 24 16:42:58.878404 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.878387    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6a0daaab-0ecb-4e33-bc7c-c53caef203f2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t4xnt\" (UID: \"6a0daaab-0ecb-4e33-bc7c-c53caef203f2\") " pod="openshift-multus/multus-additional-cni-plugins-t4xnt"
Jul 24 16:42:58.878488 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.878413    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6597s\" (UniqueName: \"kubernetes.io/projected/6a0daaab-0ecb-4e33-bc7c-c53caef203f2-kube-api-access-6597s\") pod \"multus-additional-cni-plugins-t4xnt\" (UID: \"6a0daaab-0ecb-4e33-bc7c-c53caef203f2\") " pod="openshift-multus/multus-additional-cni-plugins-t4xnt"
Jul 24 16:42:58.878488 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.878434    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/52dcf805-dee8-447d-a8d9-221be23de2d8-lib-modules\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.878488 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.878451    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6a0daaab-0ecb-4e33-bc7c-c53caef203f2-os-release\") pod \"multus-additional-cni-plugins-t4xnt\" (UID: \"6a0daaab-0ecb-4e33-bc7c-c53caef203f2\") " pod="openshift-multus/multus-additional-cni-plugins-t4xnt"
Jul 24 16:42:58.878488 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.878467    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52dcf805-dee8-447d-a8d9-221be23de2d8-tmp\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.878648 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.878490    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7vbv\" (UniqueName: \"kubernetes.io/projected/f263630d-88c9-4b5d-8afb-7075a1118555-kube-api-access-k7vbv\") pod \"node-resolver-g5sl6\" (UID: \"f263630d-88c9-4b5d-8afb-7075a1118555\") " pod="openshift-dns/node-resolver-g5sl6"
Jul 24 16:42:58.878648 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.878514    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-host-run-netns\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.878648 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.878537    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-host-run-ovn-kubernetes\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.878648 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.878549    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6a0daaab-0ecb-4e33-bc7c-c53caef203f2-os-release\") pod \"multus-additional-cni-plugins-t4xnt\" (UID: \"6a0daaab-0ecb-4e33-bc7c-c53caef203f2\") " pod="openshift-multus/multus-additional-cni-plugins-t4xnt"
Jul 24 16:42:58.878648 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.878560    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/97b382f6-038a-43cc-8db6-3a08cf2d7be2-env-overrides\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.878648 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.878600    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/573b3eaf-dd83-456c-b7e0-35cb5aad6884-kubelet-config\") pod \"global-pull-secret-syncer-vzbnm\" (UID: \"573b3eaf-dd83-456c-b7e0-35cb5aad6884\") " pod="kube-system/global-pull-secret-syncer-vzbnm"
Jul 24 16:42:58.878648 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.878625    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-host-cni-bin\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.878648 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.878643    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmzk2\" (UniqueName: \"kubernetes.io/projected/e620ec4e-9703-45c1-9080-d17039c0d2e8-kube-api-access-qmzk2\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.878956 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.878666    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce9a1972-6a1e-4e0d-9dab-0e25820dedf1-host\") pod \"node-ca-vq4cx\" (UID: \"ce9a1972-6a1e-4e0d-9dab-0e25820dedf1\") " pod="openshift-image-registry/node-ca-vq4cx"
Jul 24 16:42:58.878956 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.878694    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a0daaab-0ecb-4e33-bc7c-c53caef203f2-system-cni-dir\") pod \"multus-additional-cni-plugins-t4xnt\" (UID: \"6a0daaab-0ecb-4e33-bc7c-c53caef203f2\") " pod="openshift-multus/multus-additional-cni-plugins-t4xnt"
Jul 24 16:42:58.878956 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.878700    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce9a1972-6a1e-4e0d-9dab-0e25820dedf1-host\") pod \"node-ca-vq4cx\" (UID: \"ce9a1972-6a1e-4e0d-9dab-0e25820dedf1\") " pod="openshift-image-registry/node-ca-vq4cx"
Jul 24 16:42:58.878956 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.878662    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/573b3eaf-dd83-456c-b7e0-35cb5aad6884-kubelet-config\") pod \"global-pull-secret-syncer-vzbnm\" (UID: \"573b3eaf-dd83-456c-b7e0-35cb5aad6884\") " pod="kube-system/global-pull-secret-syncer-vzbnm"
Jul 24 16:42:58.878956 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.878718    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a0daaab-0ecb-4e33-bc7c-c53caef203f2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t4xnt\" (UID: \"6a0daaab-0ecb-4e33-bc7c-c53caef203f2\") " pod="openshift-multus/multus-additional-cni-plugins-t4xnt"
Jul 24 16:42:58.878956 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.878744    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/855016a9-43a6-4765-815e-9739917af352-agent-certs\") pod \"konnectivity-agent-45v2q\" (UID: \"855016a9-43a6-4765-815e-9739917af352\") " pod="kube-system/konnectivity-agent-45v2q"
Jul 24 16:42:58.878956 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.878752    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a0daaab-0ecb-4e33-bc7c-c53caef203f2-system-cni-dir\") pod \"multus-additional-cni-plugins-t4xnt\" (UID: \"6a0daaab-0ecb-4e33-bc7c-c53caef203f2\") " pod="openshift-multus/multus-additional-cni-plugins-t4xnt"
Jul 24 16:42:58.878956 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.878773    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52dcf805-dee8-447d-a8d9-221be23de2d8-var-lib-kubelet\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.878956 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.878797    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/97b382f6-038a-43cc-8db6-3a08cf2d7be2-ovn-node-metrics-cert\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.878956 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.878833    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aa45a2eb-0246-4044-85cb-f09d0750d858-host-slash\") pod \"iptables-alerter-gt8zs\" (UID: \"aa45a2eb-0246-4044-85cb-f09d0750d858\") " pod="openshift-network-operator/iptables-alerter-gt8zs"
Jul 24 16:42:58.878956 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.878871    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e620ec4e-9703-45c1-9080-d17039c0d2e8-cni-binary-copy\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.878956 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.878902    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/52dcf805-dee8-447d-a8d9-221be23de2d8-host\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.878956 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.878930    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/97b382f6-038a-43cc-8db6-3a08cf2d7be2-ovnkube-config\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.878956 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.878944    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a0daaab-0ecb-4e33-bc7c-c53caef203f2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t4xnt\" (UID: \"6a0daaab-0ecb-4e33-bc7c-c53caef203f2\") " pod="openshift-multus/multus-additional-cni-plugins-t4xnt"
Jul 24 16:42:58.879674 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.878986    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/52dcf805-dee8-447d-a8d9-221be23de2d8-etc-systemd\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.879674 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.879021    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-multus-socket-dir-parent\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.879674 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.878996    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aa45a2eb-0246-4044-85cb-f09d0750d858-host-slash\") pod \"iptables-alerter-gt8zs\" (UID: \"aa45a2eb-0246-4044-85cb-f09d0750d858\") " pod="openshift-network-operator/iptables-alerter-gt8zs"
Jul 24 16:42:58.879674 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.879049    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a0daaab-0ecb-4e33-bc7c-c53caef203f2-cnibin\") pod \"multus-additional-cni-plugins-t4xnt\" (UID: \"6a0daaab-0ecb-4e33-bc7c-c53caef203f2\") " pod="openshift-multus/multus-additional-cni-plugins-t4xnt"
Jul 24 16:42:58.879674 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.879076    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3c86fd31-5d0f-4814-8cb6-825111d435d7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nfw5x\" (UID: \"3c86fd31-5d0f-4814-8cb6-825111d435d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfw5x"
Jul 24 16:42:58.879674 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.879085    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 24 16:42:58.879674 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.879105    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3c86fd31-5d0f-4814-8cb6-825111d435d7-registration-dir\") pod \"aws-ebs-csi-driver-node-nfw5x\" (UID: \"3c86fd31-5d0f-4814-8cb6-825111d435d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfw5x"
Jul 24 16:42:58.879674 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.879109    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a0daaab-0ecb-4e33-bc7c-c53caef203f2-cnibin\") pod \"multus-additional-cni-plugins-t4xnt\" (UID: \"6a0daaab-0ecb-4e33-bc7c-c53caef203f2\") " pod="openshift-multus/multus-additional-cni-plugins-t4xnt"
Jul 24 16:42:58.879674 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.879138    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/52dcf805-dee8-447d-a8d9-221be23de2d8-etc-sysconfig\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.879674 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.879090    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6a0daaab-0ecb-4e33-bc7c-c53caef203f2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t4xnt\" (UID: \"6a0daaab-0ecb-4e33-bc7c-c53caef203f2\") " pod="openshift-multus/multus-additional-cni-plugins-t4xnt"
Jul 24 16:42:58.879674 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.879196    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f263630d-88c9-4b5d-8afb-7075a1118555-hosts-file\") pod \"node-resolver-g5sl6\" (UID: \"f263630d-88c9-4b5d-8afb-7075a1118555\") " pod="openshift-dns/node-resolver-g5sl6"
Jul 24 16:42:58.879674 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.879211    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-multus-socket-dir-parent\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.879674 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.879258    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-run-openvswitch\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.879674 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.879344    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-cnibin\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.879674 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.879369    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-host-run-multus-certs\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.879674 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.879387    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndwn5\" (UniqueName: \"kubernetes.io/projected/a12b44d3-5fd7-4543-bf4b-6b9ba880c11b-kube-api-access-ndwn5\") pod \"network-check-target-hhvqg\" (UID: \"a12b44d3-5fd7-4543-bf4b-6b9ba880c11b\") " pod="openshift-network-diagnostics/network-check-target-hhvqg"
Jul 24 16:42:58.879674 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.879403    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2m6p\" (UniqueName: \"kubernetes.io/projected/54f4b6d0-be12-40ae-b3ac-f7acb8e715b9-kube-api-access-r2m6p\") pod \"network-metrics-daemon-8lwz8\" (UID: \"54f4b6d0-be12-40ae-b3ac-f7acb8e715b9\") " pod="openshift-multus/network-metrics-daemon-8lwz8"
Jul 24 16:42:58.880403 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.879423    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6a0daaab-0ecb-4e33-bc7c-c53caef203f2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t4xnt\" (UID: \"6a0daaab-0ecb-4e33-bc7c-c53caef203f2\") " pod="openshift-multus/multus-additional-cni-plugins-t4xnt"
Jul 24 16:42:58.880403 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.879452    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-host-slash\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.880403 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.879461    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e620ec4e-9703-45c1-9080-d17039c0d2e8-cni-binary-copy\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.880403 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.879478    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.880403 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.879456    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-host-run-multus-certs\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.880403 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.879508    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-cnibin\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.880403 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.879529    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-multus-cni-dir\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.880403 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.879583    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-host-var-lib-cni-bin\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.880403 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.879614    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3c86fd31-5d0f-4814-8cb6-825111d435d7-device-dir\") pod \"aws-ebs-csi-driver-node-nfw5x\" (UID: \"3c86fd31-5d0f-4814-8cb6-825111d435d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfw5x"
Jul 24 16:42:58.880403 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.879588    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-multus-cni-dir\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.880403 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.879617    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-host-var-lib-cni-bin\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.880403 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.879640    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/52dcf805-dee8-447d-a8d9-221be23de2d8-etc-modprobe-d\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.880403 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.879667    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-host-kubelet\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.880403 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.879692    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/97b382f6-038a-43cc-8db6-3a08cf2d7be2-ovnkube-script-lib\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.880403 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.879719    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-system-cni-dir\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.880403 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.879746    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/855016a9-43a6-4765-815e-9739917af352-konnectivity-ca\") pod \"konnectivity-agent-45v2q\" (UID: \"855016a9-43a6-4765-815e-9739917af352\") " pod="kube-system/konnectivity-agent-45v2q"
Jul 24 16:42:58.880403 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.879770    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52dcf805-dee8-447d-a8d9-221be23de2d8-etc-kubernetes\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.881245 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.879793    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/52dcf805-dee8-447d-a8d9-221be23de2d8-etc-tuned\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.881245 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.879817    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-run-systemd\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.881245 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.879841    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-system-cni-dir\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.881245 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.879879    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-etc-openvswitch\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.881245 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.879909    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3c86fd31-5d0f-4814-8cb6-825111d435d7-socket-dir\") pod \"aws-ebs-csi-driver-node-nfw5x\" (UID: \"3c86fd31-5d0f-4814-8cb6-825111d435d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfw5x"
Jul 24 16:42:58.881245 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.879934    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-var-lib-openvswitch\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.881245 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.879981    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-host-var-lib-cni-multus\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.881245 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880025    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-host-var-lib-kubelet\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.881245 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880047    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-host-var-lib-cni-multus\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.881245 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880052    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/573b3eaf-dd83-456c-b7e0-35cb5aad6884-original-pull-secret\") pod \"global-pull-secret-syncer-vzbnm\" (UID: \"573b3eaf-dd83-456c-b7e0-35cb5aad6884\") " pod="kube-system/global-pull-secret-syncer-vzbnm"
Jul 24 16:42:58.881245 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880078    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a0daaab-0ecb-4e33-bc7c-c53caef203f2-cni-binary-copy\") pod \"multus-additional-cni-plugins-t4xnt\" (UID: \"6a0daaab-0ecb-4e33-bc7c-c53caef203f2\") " pod="openshift-multus/multus-additional-cni-plugins-t4xnt"
Jul 24 16:42:58.881245 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880095    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/52dcf805-dee8-447d-a8d9-221be23de2d8-etc-sysctl-d\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.881245 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880110    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-node-log\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.881245 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880119    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-host-var-lib-kubelet\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.881245 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880127    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-host-run-k8s-cni-cncf-io\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.881245 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:58.880135    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 16:42:58.881245 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880148    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-host-run-k8s-cni-cncf-io\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.881245 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880157    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-host-run-netns\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.881774 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880188    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-hostroot\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.881774 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880186    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-host-run-netns\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.881774 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:58.880215    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/573b3eaf-dd83-456c-b7e0-35cb5aad6884-original-pull-secret podName:573b3eaf-dd83-456c-b7e0-35cb5aad6884 nodeName:}" failed. No retries permitted until 2026-07-24 16:42:59.380186815 +0000 UTC m=+2.022150704 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/573b3eaf-dd83-456c-b7e0-35cb5aad6884-original-pull-secret") pod "global-pull-secret-syncer-vzbnm" (UID: "573b3eaf-dd83-456c-b7e0-35cb5aad6884") : object "kube-system"/"original-pull-secret" not registered
Jul 24 16:42:58.881774 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880228    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-hostroot\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.881774 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880273    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e620ec4e-9703-45c1-9080-d17039c0d2e8-multus-daemon-config\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.881774 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880300    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-host-cni-netd\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.881774 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880325    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcx7q\" (UniqueName: \"kubernetes.io/projected/3c86fd31-5d0f-4814-8cb6-825111d435d7-kube-api-access-tcx7q\") pod \"aws-ebs-csi-driver-node-nfw5x\" (UID: \"3c86fd31-5d0f-4814-8cb6-825111d435d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfw5x"
Jul 24 16:42:58.881774 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880351    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/52dcf805-dee8-447d-a8d9-221be23de2d8-run\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.881774 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880368    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/855016a9-43a6-4765-815e-9739917af352-konnectivity-ca\") pod \"konnectivity-agent-45v2q\" (UID: \"855016a9-43a6-4765-815e-9739917af352\") " pod="kube-system/konnectivity-agent-45v2q"
Jul 24 16:42:58.881774 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880377    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-os-release\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.881774 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880402    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-etc-kubernetes\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.881774 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880425    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3c86fd31-5d0f-4814-8cb6-825111d435d7-etc-selinux\") pod \"aws-ebs-csi-driver-node-nfw5x\" (UID: \"3c86fd31-5d0f-4814-8cb6-825111d435d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfw5x"
Jul 24 16:42:58.881774 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880446    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmkqv\" (UniqueName: \"kubernetes.io/projected/52dcf805-dee8-447d-a8d9-221be23de2d8-kube-api-access-pmkqv\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.881774 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880470    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/aa45a2eb-0246-4044-85cb-f09d0750d858-iptables-alerter-script\") pod \"iptables-alerter-gt8zs\" (UID: \"aa45a2eb-0246-4044-85cb-f09d0750d858\") " pod="openshift-network-operator/iptables-alerter-gt8zs"
Jul 24 16:42:58.881774 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880473    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-etc-kubernetes\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.881774 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880448    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-os-release\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.881774 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880490    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52dcf805-dee8-447d-a8d9-221be23de2d8-sys\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.882321 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880528    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-run-ovn\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.882321 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880567    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kst4t\" (UniqueName: \"kubernetes.io/projected/ce9a1972-6a1e-4e0d-9dab-0e25820dedf1-kube-api-access-kst4t\") pod \"node-ca-vq4cx\" (UID: \"ce9a1972-6a1e-4e0d-9dab-0e25820dedf1\") " pod="openshift-image-registry/node-ca-vq4cx"
Jul 24 16:42:58.882321 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880567    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a0daaab-0ecb-4e33-bc7c-c53caef203f2-cni-binary-copy\") pod \"multus-additional-cni-plugins-t4xnt\" (UID: \"6a0daaab-0ecb-4e33-bc7c-c53caef203f2\") " pod="openshift-multus/multus-additional-cni-plugins-t4xnt"
Jul 24 16:42:58.882321 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880618    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3c86fd31-5d0f-4814-8cb6-825111d435d7-sys-fs\") pod \"aws-ebs-csi-driver-node-nfw5x\" (UID: \"3c86fd31-5d0f-4814-8cb6-825111d435d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfw5x"
Jul 24 16:42:58.882321 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880640    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e620ec4e-9703-45c1-9080-d17039c0d2e8-multus-daemon-config\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.882321 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880659    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/52dcf805-dee8-447d-a8d9-221be23de2d8-etc-sysctl-conf\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.882321 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880701    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f263630d-88c9-4b5d-8afb-7075a1118555-tmp-dir\") pod \"node-resolver-g5sl6\" (UID: \"f263630d-88c9-4b5d-8afb-7075a1118555\") " pod="openshift-dns/node-resolver-g5sl6"
Jul 24 16:42:58.882321 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880728    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-systemd-units\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.882321 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880754    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-log-socket\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.882321 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880805    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pft9c\" (UniqueName: \"kubernetes.io/projected/97b382f6-038a-43cc-8db6-3a08cf2d7be2-kube-api-access-pft9c\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.882321 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880839    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzm9x\" (UniqueName: \"kubernetes.io/projected/aa45a2eb-0246-4044-85cb-f09d0750d858-kube-api-access-bzm9x\") pod \"iptables-alerter-gt8zs\" (UID: \"aa45a2eb-0246-4044-85cb-f09d0750d858\") " pod="openshift-network-operator/iptables-alerter-gt8zs"
Jul 24 16:42:58.882321 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880865    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-multus-conf-dir\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.882321 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880881    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/aa45a2eb-0246-4044-85cb-f09d0750d858-iptables-alerter-script\") pod \"iptables-alerter-gt8zs\" (UID: \"aa45a2eb-0246-4044-85cb-f09d0750d858\") " pod="openshift-network-operator/iptables-alerter-gt8zs"
Jul 24 16:42:58.882321 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880892    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/573b3eaf-dd83-456c-b7e0-35cb5aad6884-dbus\") pod \"global-pull-secret-syncer-vzbnm\" (UID: \"573b3eaf-dd83-456c-b7e0-35cb5aad6884\") " pod="kube-system/global-pull-secret-syncer-vzbnm"
Jul 24 16:42:58.882321 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880895    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6a0daaab-0ecb-4e33-bc7c-c53caef203f2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t4xnt\" (UID: \"6a0daaab-0ecb-4e33-bc7c-c53caef203f2\") " pod="openshift-multus/multus-additional-cni-plugins-t4xnt"
Jul 24 16:42:58.882321 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880919    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54f4b6d0-be12-40ae-b3ac-f7acb8e715b9-metrics-certs\") pod \"network-metrics-daemon-8lwz8\" (UID: \"54f4b6d0-be12-40ae-b3ac-f7acb8e715b9\") " pod="openshift-multus/network-metrics-daemon-8lwz8"
Jul 24 16:42:58.882321 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880945    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ce9a1972-6a1e-4e0d-9dab-0e25820dedf1-serviceca\") pod \"node-ca-vq4cx\" (UID: \"ce9a1972-6a1e-4e0d-9dab-0e25820dedf1\") " pod="openshift-image-registry/node-ca-vq4cx"
Jul 24 16:42:58.882837 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.880953    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e620ec4e-9703-45c1-9080-d17039c0d2e8-multus-conf-dir\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.882837 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.881034    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/573b3eaf-dd83-456c-b7e0-35cb5aad6884-dbus\") pod \"global-pull-secret-syncer-vzbnm\" (UID: \"573b3eaf-dd83-456c-b7e0-35cb5aad6884\") " pod="kube-system/global-pull-secret-syncer-vzbnm"
Jul 24 16:42:58.882837 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:58.881091    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 16:42:58.882837 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:58.881150    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54f4b6d0-be12-40ae-b3ac-f7acb8e715b9-metrics-certs podName:54f4b6d0-be12-40ae-b3ac-f7acb8e715b9 nodeName:}" failed. No retries permitted until 2026-07-24 16:42:59.381132704 +0000 UTC m=+2.023096582 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54f4b6d0-be12-40ae-b3ac-f7acb8e715b9-metrics-certs") pod "network-metrics-daemon-8lwz8" (UID: "54f4b6d0-be12-40ae-b3ac-f7acb8e715b9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 16:42:58.882837 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.881337    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ce9a1972-6a1e-4e0d-9dab-0e25820dedf1-serviceca\") pod \"node-ca-vq4cx\" (UID: \"ce9a1972-6a1e-4e0d-9dab-0e25820dedf1\") " pod="openshift-image-registry/node-ca-vq4cx"
Jul 24 16:42:58.882994 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.882934    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/855016a9-43a6-4765-815e-9739917af352-agent-certs\") pod \"konnectivity-agent-45v2q\" (UID: \"855016a9-43a6-4765-815e-9739917af352\") " pod="kube-system/konnectivity-agent-45v2q"
Jul 24 16:42:58.885618 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:58.885518    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 16:42:58.885618 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:58.885543    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 16:42:58.885618 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:58.885557    2563 projected.go:196] Error preparing data for projected volume kube-api-access-ndwn5 for pod openshift-network-diagnostics/network-check-target-hhvqg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 16:42:58.885618 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:58.885608    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a12b44d3-5fd7-4543-bf4b-6b9ba880c11b-kube-api-access-ndwn5 podName:a12b44d3-5fd7-4543-bf4b-6b9ba880c11b nodeName:}" failed. No retries permitted until 2026-07-24 16:42:59.385591895 +0000 UTC m=+2.027555781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ndwn5" (UniqueName: "kubernetes.io/projected/a12b44d3-5fd7-4543-bf4b-6b9ba880c11b-kube-api-access-ndwn5") pod "network-check-target-hhvqg" (UID: "a12b44d3-5fd7-4543-bf4b-6b9ba880c11b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 16:42:58.886968 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.886941    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6597s\" (UniqueName: \"kubernetes.io/projected/6a0daaab-0ecb-4e33-bc7c-c53caef203f2-kube-api-access-6597s\") pod \"multus-additional-cni-plugins-t4xnt\" (UID: \"6a0daaab-0ecb-4e33-bc7c-c53caef203f2\") " pod="openshift-multus/multus-additional-cni-plugins-t4xnt"
Jul 24 16:42:58.887386 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.887357    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2m6p\" (UniqueName: \"kubernetes.io/projected/54f4b6d0-be12-40ae-b3ac-f7acb8e715b9-kube-api-access-r2m6p\") pod \"network-metrics-daemon-8lwz8\" (UID: \"54f4b6d0-be12-40ae-b3ac-f7acb8e715b9\") " pod="openshift-multus/network-metrics-daemon-8lwz8"
Jul 24 16:42:58.887743 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.887724    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmzk2\" (UniqueName: \"kubernetes.io/projected/e620ec4e-9703-45c1-9080-d17039c0d2e8-kube-api-access-qmzk2\") pod \"multus-dblpg\" (UID: \"e620ec4e-9703-45c1-9080-d17039c0d2e8\") " pod="openshift-multus/multus-dblpg"
Jul 24 16:42:58.888904 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.888877    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kst4t\" (UniqueName: \"kubernetes.io/projected/ce9a1972-6a1e-4e0d-9dab-0e25820dedf1-kube-api-access-kst4t\") pod \"node-ca-vq4cx\" (UID: \"ce9a1972-6a1e-4e0d-9dab-0e25820dedf1\") " pod="openshift-image-registry/node-ca-vq4cx"
Jul 24 16:42:58.889168 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.889146    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzm9x\" (UniqueName: \"kubernetes.io/projected/aa45a2eb-0246-4044-85cb-f09d0750d858-kube-api-access-bzm9x\") pod \"iptables-alerter-gt8zs\" (UID: \"aa45a2eb-0246-4044-85cb-f09d0750d858\") " pod="openshift-network-operator/iptables-alerter-gt8zs"
Jul 24 16:42:58.940523 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.940472    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 16:37:58 +0000 UTC" deadline="2028-01-07 03:07:19.767764172 +0000 UTC"
Jul 24 16:42:58.940523 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.940521    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12754h24m20.827247844s"
Jul 24 16:42:58.954004 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.953984    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 16:42:58.981232 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.981206    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-run-ovn\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.981424 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.981244    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3c86fd31-5d0f-4814-8cb6-825111d435d7-sys-fs\") pod \"aws-ebs-csi-driver-node-nfw5x\" (UID: \"3c86fd31-5d0f-4814-8cb6-825111d435d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfw5x"
Jul 24 16:42:58.981424 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.981262    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/52dcf805-dee8-447d-a8d9-221be23de2d8-etc-sysctl-conf\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.981424 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.981295    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f263630d-88c9-4b5d-8afb-7075a1118555-tmp-dir\") pod \"node-resolver-g5sl6\" (UID: \"f263630d-88c9-4b5d-8afb-7075a1118555\") " pod="openshift-dns/node-resolver-g5sl6"
Jul 24 16:42:58.981424 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.981311    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-systemd-units\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.981424 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.981323    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-run-ovn\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.981424 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.981390    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-systemd-units\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.981424 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.981402    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-log-socket\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.981424 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.981328    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-log-socket\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.981810 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.981447    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pft9c\" (UniqueName: \"kubernetes.io/projected/97b382f6-038a-43cc-8db6-3a08cf2d7be2-kube-api-access-pft9c\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.981810 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.981455    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3c86fd31-5d0f-4814-8cb6-825111d435d7-sys-fs\") pod \"aws-ebs-csi-driver-node-nfw5x\" (UID: \"3c86fd31-5d0f-4814-8cb6-825111d435d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfw5x"
Jul 24 16:42:58.981810 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.981495    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/52dcf805-dee8-447d-a8d9-221be23de2d8-lib-modules\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.981810 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.981504    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/52dcf805-dee8-447d-a8d9-221be23de2d8-etc-sysctl-conf\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.981810 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.981522    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52dcf805-dee8-447d-a8d9-221be23de2d8-tmp\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.981810 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.981547    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7vbv\" (UniqueName: \"kubernetes.io/projected/f263630d-88c9-4b5d-8afb-7075a1118555-kube-api-access-k7vbv\") pod \"node-resolver-g5sl6\" (UID: \"f263630d-88c9-4b5d-8afb-7075a1118555\") " pod="openshift-dns/node-resolver-g5sl6"
Jul 24 16:42:58.981810 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.981569    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-host-run-netns\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.981810 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.981593    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-host-run-ovn-kubernetes\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.981810 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.981612    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/52dcf805-dee8-447d-a8d9-221be23de2d8-lib-modules\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.981810 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.981618    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/97b382f6-038a-43cc-8db6-3a08cf2d7be2-env-overrides\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.981810 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.981643    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-host-cni-bin\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.981810 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.981653    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-host-run-netns\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.981810 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.981669    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f263630d-88c9-4b5d-8afb-7075a1118555-tmp-dir\") pod \"node-resolver-g5sl6\" (UID: \"f263630d-88c9-4b5d-8afb-7075a1118555\") " pod="openshift-dns/node-resolver-g5sl6"
Jul 24 16:42:58.981810 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.981674    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52dcf805-dee8-447d-a8d9-221be23de2d8-var-lib-kubelet\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.981810 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.981715    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/97b382f6-038a-43cc-8db6-3a08cf2d7be2-ovn-node-metrics-cert\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.981810 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.981725    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52dcf805-dee8-447d-a8d9-221be23de2d8-var-lib-kubelet\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.981810 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.981745    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/52dcf805-dee8-447d-a8d9-221be23de2d8-host\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.981810 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.981770    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/97b382f6-038a-43cc-8db6-3a08cf2d7be2-ovnkube-config\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.982843 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.981751    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-host-run-ovn-kubernetes\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.982843 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.981798    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/52dcf805-dee8-447d-a8d9-221be23de2d8-etc-systemd\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.982843 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.981817    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-host-cni-bin\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.982843 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.981827    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3c86fd31-5d0f-4814-8cb6-825111d435d7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nfw5x\" (UID: \"3c86fd31-5d0f-4814-8cb6-825111d435d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfw5x"
Jul 24 16:42:58.982843 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.981891    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/52dcf805-dee8-447d-a8d9-221be23de2d8-host\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.982843 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.981899    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3c86fd31-5d0f-4814-8cb6-825111d435d7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nfw5x\" (UID: \"3c86fd31-5d0f-4814-8cb6-825111d435d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfw5x"
Jul 24 16:42:58.982843 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.981899    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3c86fd31-5d0f-4814-8cb6-825111d435d7-registration-dir\") pod \"aws-ebs-csi-driver-node-nfw5x\" (UID: \"3c86fd31-5d0f-4814-8cb6-825111d435d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfw5x"
Jul 24 16:42:58.982843 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.981938    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/52dcf805-dee8-447d-a8d9-221be23de2d8-etc-sysconfig\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.982843 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.981992    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f263630d-88c9-4b5d-8afb-7075a1118555-hosts-file\") pod \"node-resolver-g5sl6\" (UID: \"f263630d-88c9-4b5d-8afb-7075a1118555\") " pod="openshift-dns/node-resolver-g5sl6"
Jul 24 16:42:58.982843 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.981951    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3c86fd31-5d0f-4814-8cb6-825111d435d7-registration-dir\") pod \"aws-ebs-csi-driver-node-nfw5x\" (UID: \"3c86fd31-5d0f-4814-8cb6-825111d435d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfw5x"
Jul 24 16:42:58.982843 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982043    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-run-openvswitch\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.982843 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982053    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f263630d-88c9-4b5d-8afb-7075a1118555-hosts-file\") pod \"node-resolver-g5sl6\" (UID: \"f263630d-88c9-4b5d-8afb-7075a1118555\") " pod="openshift-dns/node-resolver-g5sl6"
Jul 24 16:42:58.982843 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.981945    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/52dcf805-dee8-447d-a8d9-221be23de2d8-etc-systemd\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.982843 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982093    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-host-slash\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.982843 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982102    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/52dcf805-dee8-447d-a8d9-221be23de2d8-etc-sysconfig\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.982843 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982128    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.982843 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982137    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-run-openvswitch\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.983793 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982174    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3c86fd31-5d0f-4814-8cb6-825111d435d7-device-dir\") pod \"aws-ebs-csi-driver-node-nfw5x\" (UID: \"3c86fd31-5d0f-4814-8cb6-825111d435d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfw5x"
Jul 24 16:42:58.983793 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982179    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.983793 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982177    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-host-slash\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.983793 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982210    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/52dcf805-dee8-447d-a8d9-221be23de2d8-etc-modprobe-d\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.983793 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982220    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3c86fd31-5d0f-4814-8cb6-825111d435d7-device-dir\") pod \"aws-ebs-csi-driver-node-nfw5x\" (UID: \"3c86fd31-5d0f-4814-8cb6-825111d435d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfw5x"
Jul 24 16:42:58.983793 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982239    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-host-kubelet\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.983793 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982265    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/97b382f6-038a-43cc-8db6-3a08cf2d7be2-ovnkube-script-lib\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.983793 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982295    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52dcf805-dee8-447d-a8d9-221be23de2d8-etc-kubernetes\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.983793 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982322    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/52dcf805-dee8-447d-a8d9-221be23de2d8-etc-tuned\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.983793 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982335    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-host-kubelet\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.983793 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982296    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/52dcf805-dee8-447d-a8d9-221be23de2d8-etc-modprobe-d\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.983793 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982394    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-run-systemd\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.983793 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982348    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-run-systemd\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.983793 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982435    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-etc-openvswitch\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.983793 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982463    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3c86fd31-5d0f-4814-8cb6-825111d435d7-socket-dir\") pod \"aws-ebs-csi-driver-node-nfw5x\" (UID: \"3c86fd31-5d0f-4814-8cb6-825111d435d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfw5x"
Jul 24 16:42:58.983793 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982479    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52dcf805-dee8-447d-a8d9-221be23de2d8-etc-kubernetes\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.983793 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982489    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-var-lib-openvswitch\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.984527 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982524    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/97b382f6-038a-43cc-8db6-3a08cf2d7be2-ovnkube-config\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.984527 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982532    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-var-lib-openvswitch\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.984527 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982539    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/52dcf805-dee8-447d-a8d9-221be23de2d8-etc-sysctl-d\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.984527 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982564    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-etc-openvswitch\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.984527 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982567    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-node-log\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.984527 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982606    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-host-cni-netd\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.984527 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982633    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcx7q\" (UniqueName: \"kubernetes.io/projected/3c86fd31-5d0f-4814-8cb6-825111d435d7-kube-api-access-tcx7q\") pod \"aws-ebs-csi-driver-node-nfw5x\" (UID: \"3c86fd31-5d0f-4814-8cb6-825111d435d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfw5x"
Jul 24 16:42:58.984527 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982663    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/52dcf805-dee8-447d-a8d9-221be23de2d8-run\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.984527 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982672    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/52dcf805-dee8-447d-a8d9-221be23de2d8-etc-sysctl-d\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.984527 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982691    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3c86fd31-5d0f-4814-8cb6-825111d435d7-etc-selinux\") pod \"aws-ebs-csi-driver-node-nfw5x\" (UID: \"3c86fd31-5d0f-4814-8cb6-825111d435d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfw5x"
Jul 24 16:42:58.984527 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982719    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-node-log\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.984527 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982720    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmkqv\" (UniqueName: \"kubernetes.io/projected/52dcf805-dee8-447d-a8d9-221be23de2d8-kube-api-access-pmkqv\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.984527 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982729    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/97b382f6-038a-43cc-8db6-3a08cf2d7be2-host-cni-netd\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.984527 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982804    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/52dcf805-dee8-447d-a8d9-221be23de2d8-run\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.984527 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982852    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3c86fd31-5d0f-4814-8cb6-825111d435d7-socket-dir\") pod \"aws-ebs-csi-driver-node-nfw5x\" (UID: \"3c86fd31-5d0f-4814-8cb6-825111d435d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfw5x"
Jul 24 16:42:58.984527 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982889    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/97b382f6-038a-43cc-8db6-3a08cf2d7be2-ovnkube-script-lib\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.984527 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982895    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52dcf805-dee8-447d-a8d9-221be23de2d8-sys\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.984527 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982892    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/97b382f6-038a-43cc-8db6-3a08cf2d7be2-env-overrides\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.985138 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.982982    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52dcf805-dee8-447d-a8d9-221be23de2d8-sys\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.985138 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.983104    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3c86fd31-5d0f-4814-8cb6-825111d435d7-etc-selinux\") pod \"aws-ebs-csi-driver-node-nfw5x\" (UID: \"3c86fd31-5d0f-4814-8cb6-825111d435d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfw5x"
Jul 24 16:42:58.985138 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.984466    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52dcf805-dee8-447d-a8d9-221be23de2d8-tmp\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.985138 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.984688    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/97b382f6-038a-43cc-8db6-3a08cf2d7be2-ovn-node-metrics-cert\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.985138 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.984780    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/52dcf805-dee8-447d-a8d9-221be23de2d8-etc-tuned\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.990138 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.990114    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcx7q\" (UniqueName: \"kubernetes.io/projected/3c86fd31-5d0f-4814-8cb6-825111d435d7-kube-api-access-tcx7q\") pod \"aws-ebs-csi-driver-node-nfw5x\" (UID: \"3c86fd31-5d0f-4814-8cb6-825111d435d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfw5x"
Jul 24 16:42:58.990411 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.990381    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pft9c\" (UniqueName: \"kubernetes.io/projected/97b382f6-038a-43cc-8db6-3a08cf2d7be2-kube-api-access-pft9c\") pod \"ovnkube-node-gt5dk\" (UID: \"97b382f6-038a-43cc-8db6-3a08cf2d7be2\") " pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:58.990589 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.990573    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmkqv\" (UniqueName: \"kubernetes.io/projected/52dcf805-dee8-447d-a8d9-221be23de2d8-kube-api-access-pmkqv\") pod \"tuned-mvlt2\" (UID: \"52dcf805-dee8-447d-a8d9-221be23de2d8\") " pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:58.990666 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:58.990642    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7vbv\" (UniqueName: \"kubernetes.io/projected/f263630d-88c9-4b5d-8afb-7075a1118555-kube-api-access-k7vbv\") pod \"node-resolver-g5sl6\" (UID: \"f263630d-88c9-4b5d-8afb-7075a1118555\") " pod="openshift-dns/node-resolver-g5sl6"
Jul 24 16:42:59.075408 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:59.075378    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vq4cx"
Jul 24 16:42:59.091260 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:59.091211    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t4xnt"
Jul 24 16:42:59.094073 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:59.094040    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cdfb9b825abe617ad3cdd7a8d779a3b.slice/crio-8c381950b4f3f7d8ddbe6e75654242d34be3e246188f6ff501f93d1b70c42a46 WatchSource:0}: Error finding container 8c381950b4f3f7d8ddbe6e75654242d34be3e246188f6ff501f93d1b70c42a46: Status 404 returned error can't find the container with id 8c381950b4f3f7d8ddbe6e75654242d34be3e246188f6ff501f93d1b70c42a46
Jul 24 16:42:59.100095 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:59.100000    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce9a1972_6a1e_4e0d_9dab_0e25820dedf1.slice/crio-6465e6bc71f995a88c018ca4af2fe6e895bf85d52a099e71bc199ef1f0f8bd0d WatchSource:0}: Error finding container 6465e6bc71f995a88c018ca4af2fe6e895bf85d52a099e71bc199ef1f0f8bd0d: Status 404 returned error can't find the container with id 6465e6bc71f995a88c018ca4af2fe6e895bf85d52a099e71bc199ef1f0f8bd0d
Jul 24 16:42:59.100319 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:59.100299    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 16:42:59.103229 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:59.103152    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0daaab_0ecb_4e33_bc7c_c53caef203f2.slice/crio-c7fcb9ed4c69ed6072671b84cf0c08b81c0203c8fa30f820aa47800e36a1317c WatchSource:0}: Error finding container c7fcb9ed4c69ed6072671b84cf0c08b81c0203c8fa30f820aa47800e36a1317c: Status 404 returned error can't find the container with id c7fcb9ed4c69ed6072671b84cf0c08b81c0203c8fa30f820aa47800e36a1317c
Jul 24 16:42:59.103964 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:59.103934    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5737b582bc5c83055d2673f557e5384a.slice/crio-8a7e3e8fa8f8411e385b8f6a9be98965435b1565fbec7bdfc1f81d72082e3d3f WatchSource:0}: Error finding container 8a7e3e8fa8f8411e385b8f6a9be98965435b1565fbec7bdfc1f81d72082e3d3f: Status 404 returned error can't find the container with id 8a7e3e8fa8f8411e385b8f6a9be98965435b1565fbec7bdfc1f81d72082e3d3f
Jul 24 16:42:59.120100 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:59.120077    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dblpg"
Jul 24 16:42:59.132337 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:59.132313    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode620ec4e_9703_45c1_9080_d17039c0d2e8.slice/crio-6cfdba7094a3441412cd80034ef17b5ecb1b65d7f5b5f12fb959bd901187df2b WatchSource:0}: Error finding container 6cfdba7094a3441412cd80034ef17b5ecb1b65d7f5b5f12fb959bd901187df2b: Status 404 returned error can't find the container with id 6cfdba7094a3441412cd80034ef17b5ecb1b65d7f5b5f12fb959bd901187df2b
Jul 24 16:42:59.135493 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:59.135475    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-45v2q"
Jul 24 16:42:59.143436 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:59.143416    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod855016a9_43a6_4765_815e_9739917af352.slice/crio-b56c2ee9fc1ad3245cbabc86e12a2f0d05c868baad278f1ece73aa83b6d26c31 WatchSource:0}: Error finding container b56c2ee9fc1ad3245cbabc86e12a2f0d05c868baad278f1ece73aa83b6d26c31: Status 404 returned error can't find the container with id b56c2ee9fc1ad3245cbabc86e12a2f0d05c868baad278f1ece73aa83b6d26c31
Jul 24 16:42:59.157068 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:59.157034    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gt8zs"
Jul 24 16:42:59.163166 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:59.163146    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:42:59.164785 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:59.164753    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa45a2eb_0246_4044_85cb_f09d0750d858.slice/crio-32c2929c748158a458504b0044135a3d2ee2755c4a45923c2ee9389bfcee540f WatchSource:0}: Error finding container 32c2929c748158a458504b0044135a3d2ee2755c4a45923c2ee9389bfcee540f: Status 404 returned error can't find the container with id 32c2929c748158a458504b0044135a3d2ee2755c4a45923c2ee9389bfcee540f
Jul 24 16:42:59.170897 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:59.170875    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b382f6_038a_43cc_8db6_3a08cf2d7be2.slice/crio-bd4489756d119e0cdfdbecc97ce278825e775180bbe214df21bc67629d7d0fcd WatchSource:0}: Error finding container bd4489756d119e0cdfdbecc97ce278825e775180bbe214df21bc67629d7d0fcd: Status 404 returned error can't find the container with id bd4489756d119e0cdfdbecc97ce278825e775180bbe214df21bc67629d7d0fcd
Jul 24 16:42:59.198124 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:59.198104    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfw5x"
Jul 24 16:42:59.207484 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:59.207464    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c86fd31_5d0f_4814_8cb6_825111d435d7.slice/crio-181b95598e0f5001342d145c26295f31d441e38d576ecae652ddc77372b32991 WatchSource:0}: Error finding container 181b95598e0f5001342d145c26295f31d441e38d576ecae652ddc77372b32991: Status 404 returned error can't find the container with id 181b95598e0f5001342d145c26295f31d441e38d576ecae652ddc77372b32991
Jul 24 16:42:59.211538 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:59.211520    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mvlt2"
Jul 24 16:42:59.218261 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:59.218241    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g5sl6"
Jul 24 16:42:59.220159 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:59.220139    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52dcf805_dee8_447d_a8d9_221be23de2d8.slice/crio-3d0de7abfb0f56d388937e1a4b4d50d7e7a109f40ffb07c8fea9f47fa279e1b0 WatchSource:0}: Error finding container 3d0de7abfb0f56d388937e1a4b4d50d7e7a109f40ffb07c8fea9f47fa279e1b0: Status 404 returned error can't find the container with id 3d0de7abfb0f56d388937e1a4b4d50d7e7a109f40ffb07c8fea9f47fa279e1b0
Jul 24 16:42:59.226419 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:42:59.226399    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf263630d_88c9_4b5d_8afb_7075a1118555.slice/crio-2c16251098f82fc750c6f7ee878984e8f8fcd94cd014a9d9cc3e331d3d4a9310 WatchSource:0}: Error finding container 2c16251098f82fc750c6f7ee878984e8f8fcd94cd014a9d9cc3e331d3d4a9310: Status 404 returned error can't find the container with id 2c16251098f82fc750c6f7ee878984e8f8fcd94cd014a9d9cc3e331d3d4a9310
Jul 24 16:42:59.228173 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:59.228151    2563 kuberuntime_manager.go:1459] "Unhandled Error" err="container dns-node-resolver start failed in pod node-resolver-g5sl6_openshift-dns(f263630d-88c9-4b5d-8afb-7075a1118555): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Jul 24 16:42:59.228237 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:59.228197    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"dns-node-resolver\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-dns/node-resolver-g5sl6" podUID="f263630d-88c9-4b5d-8afb-7075a1118555"
Jul 24 16:42:59.385415 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:59.385333    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54f4b6d0-be12-40ae-b3ac-f7acb8e715b9-metrics-certs\") pod \"network-metrics-daemon-8lwz8\" (UID: \"54f4b6d0-be12-40ae-b3ac-f7acb8e715b9\") " pod="openshift-multus/network-metrics-daemon-8lwz8"
Jul 24 16:42:59.385415 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:59.385394    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/573b3eaf-dd83-456c-b7e0-35cb5aad6884-original-pull-secret\") pod \"global-pull-secret-syncer-vzbnm\" (UID: \"573b3eaf-dd83-456c-b7e0-35cb5aad6884\") " pod="kube-system/global-pull-secret-syncer-vzbnm"
Jul 24 16:42:59.385602 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:59.385449    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 16:42:59.385602 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:59.385487    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 16:42:59.385602 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:59.385511    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54f4b6d0-be12-40ae-b3ac-f7acb8e715b9-metrics-certs podName:54f4b6d0-be12-40ae-b3ac-f7acb8e715b9 nodeName:}" failed. No retries permitted until 2026-07-24 16:43:00.385496281 +0000 UTC m=+3.027460145 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54f4b6d0-be12-40ae-b3ac-f7acb8e715b9-metrics-certs") pod "network-metrics-daemon-8lwz8" (UID: "54f4b6d0-be12-40ae-b3ac-f7acb8e715b9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 16:42:59.385602 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:59.385525    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/573b3eaf-dd83-456c-b7e0-35cb5aad6884-original-pull-secret podName:573b3eaf-dd83-456c-b7e0-35cb5aad6884 nodeName:}" failed. No retries permitted until 2026-07-24 16:43:00.385518619 +0000 UTC m=+3.027482484 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/573b3eaf-dd83-456c-b7e0-35cb5aad6884-original-pull-secret") pod "global-pull-secret-syncer-vzbnm" (UID: "573b3eaf-dd83-456c-b7e0-35cb5aad6884") : object "kube-system"/"original-pull-secret" not registered
Jul 24 16:42:59.486681 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:59.486651    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndwn5\" (UniqueName: \"kubernetes.io/projected/a12b44d3-5fd7-4543-bf4b-6b9ba880c11b-kube-api-access-ndwn5\") pod \"network-check-target-hhvqg\" (UID: \"a12b44d3-5fd7-4543-bf4b-6b9ba880c11b\") " pod="openshift-network-diagnostics/network-check-target-hhvqg"
Jul 24 16:42:59.486803 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:59.486776    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 16:42:59.486803 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:59.486790    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 16:42:59.486803 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:59.486799    2563 projected.go:196] Error preparing data for projected volume kube-api-access-ndwn5 for pod openshift-network-diagnostics/network-check-target-hhvqg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 16:42:59.486918 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:42:59.486842    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a12b44d3-5fd7-4543-bf4b-6b9ba880c11b-kube-api-access-ndwn5 podName:a12b44d3-5fd7-4543-bf4b-6b9ba880c11b nodeName:}" failed. No retries permitted until 2026-07-24 16:43:00.486829933 +0000 UTC m=+3.128793797 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ndwn5" (UniqueName: "kubernetes.io/projected/a12b44d3-5fd7-4543-bf4b-6b9ba880c11b-kube-api-access-ndwn5") pod "network-check-target-hhvqg" (UID: "a12b44d3-5fd7-4543-bf4b-6b9ba880c11b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 16:42:59.824655 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:59.824566    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 16:37:59 +0000 UTC" deadline="2028-03-21 06:00:23.340974535 +0000 UTC"
Jul 24 16:42:59.824655 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:42:59.824604    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14533h17m23.516374673s"
Jul 24 16:43:00.003674 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:00.002743    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hhvqg"
Jul 24 16:43:00.003674 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:00.002919    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hhvqg" podUID="a12b44d3-5fd7-4543-bf4b-6b9ba880c11b"
Jul 24 16:43:00.027482 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:00.027415    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gt8zs" event={"ID":"aa45a2eb-0246-4044-85cb-f09d0750d858","Type":"ContainerStarted","Data":"32c2929c748158a458504b0044135a3d2ee2755c4a45923c2ee9389bfcee540f"}
Jul 24 16:43:00.036906 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:00.036825    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dblpg" event={"ID":"e620ec4e-9703-45c1-9080-d17039c0d2e8","Type":"ContainerStarted","Data":"6cfdba7094a3441412cd80034ef17b5ecb1b65d7f5b5f12fb959bd901187df2b"}
Jul 24 16:43:00.047607 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:00.047525    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-234.ec2.internal" event={"ID":"5737b582bc5c83055d2673f557e5384a","Type":"ContainerStarted","Data":"8a7e3e8fa8f8411e385b8f6a9be98965435b1565fbec7bdfc1f81d72082e3d3f"}
Jul 24 16:43:00.056618 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:00.056534    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4xnt" event={"ID":"6a0daaab-0ecb-4e33-bc7c-c53caef203f2","Type":"ContainerStarted","Data":"c7fcb9ed4c69ed6072671b84cf0c08b81c0203c8fa30f820aa47800e36a1317c"}
Jul 24 16:43:00.069802 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:00.069777    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vq4cx" event={"ID":"ce9a1972-6a1e-4e0d-9dab-0e25820dedf1","Type":"ContainerStarted","Data":"6465e6bc71f995a88c018ca4af2fe6e895bf85d52a099e71bc199ef1f0f8bd0d"}
Jul 24 16:43:00.075080 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:00.075008    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal" event={"ID":"5cdfb9b825abe617ad3cdd7a8d779a3b","Type":"ContainerStarted","Data":"8c381950b4f3f7d8ddbe6e75654242d34be3e246188f6ff501f93d1b70c42a46"}
Jul 24 16:43:00.077566 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:00.077542    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 16:43:00.085886 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:00.085829    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mvlt2" event={"ID":"52dcf805-dee8-447d-a8d9-221be23de2d8","Type":"ContainerStarted","Data":"3d0de7abfb0f56d388937e1a4b4d50d7e7a109f40ffb07c8fea9f47fa279e1b0"}
Jul 24 16:43:00.093073 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:00.093047    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfw5x" event={"ID":"3c86fd31-5d0f-4814-8cb6-825111d435d7","Type":"ContainerStarted","Data":"181b95598e0f5001342d145c26295f31d441e38d576ecae652ddc77372b32991"}
Jul 24 16:43:00.100396 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:00.100368    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-45v2q" event={"ID":"855016a9-43a6-4765-815e-9739917af352","Type":"ContainerStarted","Data":"b56c2ee9fc1ad3245cbabc86e12a2f0d05c868baad278f1ece73aa83b6d26c31"}
Jul 24 16:43:00.102921 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:00.102896    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g5sl6" event={"ID":"f263630d-88c9-4b5d-8afb-7075a1118555","Type":"ContainerStarted","Data":"2c16251098f82fc750c6f7ee878984e8f8fcd94cd014a9d9cc3e331d3d4a9310"}
Jul 24 16:43:00.107084 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:00.107055    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"dns-node-resolver\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:233b9dfa42a292bdae9563a0cfb304f006bf4cf1f75229e9b3147c1019c74859\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-dns/node-resolver-g5sl6" podUID="f263630d-88c9-4b5d-8afb-7075a1118555"
Jul 24 16:43:00.119024 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:00.118987    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk" event={"ID":"97b382f6-038a-43cc-8db6-3a08cf2d7be2","Type":"ContainerStarted","Data":"bd4489756d119e0cdfdbecc97ce278825e775180bbe214df21bc67629d7d0fcd"}
Jul 24 16:43:00.407620 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:00.407530    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54f4b6d0-be12-40ae-b3ac-f7acb8e715b9-metrics-certs\") pod \"network-metrics-daemon-8lwz8\" (UID: \"54f4b6d0-be12-40ae-b3ac-f7acb8e715b9\") " pod="openshift-multus/network-metrics-daemon-8lwz8"
Jul 24 16:43:00.407796 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:00.407641    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/573b3eaf-dd83-456c-b7e0-35cb5aad6884-original-pull-secret\") pod \"global-pull-secret-syncer-vzbnm\" (UID: \"573b3eaf-dd83-456c-b7e0-35cb5aad6884\") " pod="kube-system/global-pull-secret-syncer-vzbnm"
Jul 24 16:43:00.407796 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:00.407782    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 16:43:00.407924 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:00.407849    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/573b3eaf-dd83-456c-b7e0-35cb5aad6884-original-pull-secret podName:573b3eaf-dd83-456c-b7e0-35cb5aad6884 nodeName:}" failed. No retries permitted until 2026-07-24 16:43:02.407829426 +0000 UTC m=+5.049793293 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/573b3eaf-dd83-456c-b7e0-35cb5aad6884-original-pull-secret") pod "global-pull-secret-syncer-vzbnm" (UID: "573b3eaf-dd83-456c-b7e0-35cb5aad6884") : object "kube-system"/"original-pull-secret" not registered
Jul 24 16:43:00.407990 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:00.407934    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 16:43:00.407990 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:00.407970    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54f4b6d0-be12-40ae-b3ac-f7acb8e715b9-metrics-certs podName:54f4b6d0-be12-40ae-b3ac-f7acb8e715b9 nodeName:}" failed. No retries permitted until 2026-07-24 16:43:02.407957571 +0000 UTC m=+5.049921440 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54f4b6d0-be12-40ae-b3ac-f7acb8e715b9-metrics-certs") pod "network-metrics-daemon-8lwz8" (UID: "54f4b6d0-be12-40ae-b3ac-f7acb8e715b9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 16:43:00.508340 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:00.508300    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndwn5\" (UniqueName: \"kubernetes.io/projected/a12b44d3-5fd7-4543-bf4b-6b9ba880c11b-kube-api-access-ndwn5\") pod \"network-check-target-hhvqg\" (UID: \"a12b44d3-5fd7-4543-bf4b-6b9ba880c11b\") " pod="openshift-network-diagnostics/network-check-target-hhvqg"
Jul 24 16:43:00.508547 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:00.508505    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 16:43:00.508547 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:00.508535    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 16:43:00.508674 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:00.508551    2563 projected.go:196] Error preparing data for projected volume kube-api-access-ndwn5 for pod openshift-network-diagnostics/network-check-target-hhvqg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 16:43:00.508674 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:00.508621    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a12b44d3-5fd7-4543-bf4b-6b9ba880c11b-kube-api-access-ndwn5 podName:a12b44d3-5fd7-4543-bf4b-6b9ba880c11b nodeName:}" failed. No retries permitted until 2026-07-24 16:43:02.508597019 +0000 UTC m=+5.150560902 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ndwn5" (UniqueName: "kubernetes.io/projected/a12b44d3-5fd7-4543-bf4b-6b9ba880c11b-kube-api-access-ndwn5") pod "network-check-target-hhvqg" (UID: "a12b44d3-5fd7-4543-bf4b-6b9ba880c11b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 16:43:00.824876 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:00.824761    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 16:37:59 +0000 UTC" deadline="2028-01-24 20:17:49.307382517 +0000 UTC"
Jul 24 16:43:00.824876 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:00.824828    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13179h34m48.48255906s"
Jul 24 16:43:00.996505 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:00.996465    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lwz8"
Jul 24 16:43:00.996728 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:00.996608    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lwz8" podUID="54f4b6d0-be12-40ae-b3ac-f7acb8e715b9"
Jul 24 16:43:00.996986 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:00.996884    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vzbnm"
Jul 24 16:43:00.997125 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:00.997008    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vzbnm" podUID="573b3eaf-dd83-456c-b7e0-35cb5aad6884"
Jul 24 16:43:01.998748 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:01.998699    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hhvqg"
Jul 24 16:43:01.999243 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:01.998832    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hhvqg" podUID="a12b44d3-5fd7-4543-bf4b-6b9ba880c11b"
Jul 24 16:43:02.426351 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:02.426250    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54f4b6d0-be12-40ae-b3ac-f7acb8e715b9-metrics-certs\") pod \"network-metrics-daemon-8lwz8\" (UID: \"54f4b6d0-be12-40ae-b3ac-f7acb8e715b9\") " pod="openshift-multus/network-metrics-daemon-8lwz8"
Jul 24 16:43:02.426526 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:02.426357    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/573b3eaf-dd83-456c-b7e0-35cb5aad6884-original-pull-secret\") pod \"global-pull-secret-syncer-vzbnm\" (UID: \"573b3eaf-dd83-456c-b7e0-35cb5aad6884\") " pod="kube-system/global-pull-secret-syncer-vzbnm"
Jul 24 16:43:02.426595 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:02.426529    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 16:43:02.426648 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:02.426597    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/573b3eaf-dd83-456c-b7e0-35cb5aad6884-original-pull-secret podName:573b3eaf-dd83-456c-b7e0-35cb5aad6884 nodeName:}" failed. No retries permitted until 2026-07-24 16:43:06.426577088 +0000 UTC m=+9.068540971 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/573b3eaf-dd83-456c-b7e0-35cb5aad6884-original-pull-secret") pod "global-pull-secret-syncer-vzbnm" (UID: "573b3eaf-dd83-456c-b7e0-35cb5aad6884") : object "kube-system"/"original-pull-secret" not registered
Jul 24 16:43:02.427116 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:02.427096    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 16:43:02.427233 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:02.427151    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54f4b6d0-be12-40ae-b3ac-f7acb8e715b9-metrics-certs podName:54f4b6d0-be12-40ae-b3ac-f7acb8e715b9 nodeName:}" failed. No retries permitted until 2026-07-24 16:43:06.427136945 +0000 UTC m=+9.069100824 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54f4b6d0-be12-40ae-b3ac-f7acb8e715b9-metrics-certs") pod "network-metrics-daemon-8lwz8" (UID: "54f4b6d0-be12-40ae-b3ac-f7acb8e715b9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 16:43:02.527485 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:02.527422    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndwn5\" (UniqueName: \"kubernetes.io/projected/a12b44d3-5fd7-4543-bf4b-6b9ba880c11b-kube-api-access-ndwn5\") pod \"network-check-target-hhvqg\" (UID: \"a12b44d3-5fd7-4543-bf4b-6b9ba880c11b\") " pod="openshift-network-diagnostics/network-check-target-hhvqg"
Jul 24 16:43:02.527673 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:02.527633    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 16:43:02.527673 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:02.527670    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 16:43:02.527808 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:02.527685    2563 projected.go:196] Error preparing data for projected volume kube-api-access-ndwn5 for pod openshift-network-diagnostics/network-check-target-hhvqg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 16:43:02.527808 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:02.527747    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a12b44d3-5fd7-4543-bf4b-6b9ba880c11b-kube-api-access-ndwn5 podName:a12b44d3-5fd7-4543-bf4b-6b9ba880c11b nodeName:}" failed. No retries permitted until 2026-07-24 16:43:06.527727601 +0000 UTC m=+9.169691483 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ndwn5" (UniqueName: "kubernetes.io/projected/a12b44d3-5fd7-4543-bf4b-6b9ba880c11b-kube-api-access-ndwn5") pod "network-check-target-hhvqg" (UID: "a12b44d3-5fd7-4543-bf4b-6b9ba880c11b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 16:43:02.996909 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:02.996824    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lwz8"
Jul 24 16:43:02.997097 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:02.997008    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lwz8" podUID="54f4b6d0-be12-40ae-b3ac-f7acb8e715b9"
Jul 24 16:43:02.997097 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:02.997057    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vzbnm"
Jul 24 16:43:02.997225 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:02.997179    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vzbnm" podUID="573b3eaf-dd83-456c-b7e0-35cb5aad6884"
Jul 24 16:43:03.999294 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:03.998760    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hhvqg"
Jul 24 16:43:03.999294 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:03.998899    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hhvqg" podUID="a12b44d3-5fd7-4543-bf4b-6b9ba880c11b"
Jul 24 16:43:04.996187 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:04.996145    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vzbnm"
Jul 24 16:43:04.996403 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:04.996157    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lwz8"
Jul 24 16:43:04.996403 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:04.996307    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vzbnm" podUID="573b3eaf-dd83-456c-b7e0-35cb5aad6884"
Jul 24 16:43:04.996403 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:04.996347    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lwz8" podUID="54f4b6d0-be12-40ae-b3ac-f7acb8e715b9"
Jul 24 16:43:05.996867 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:05.996641    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hhvqg"
Jul 24 16:43:05.996867 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:05.996784    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hhvqg" podUID="a12b44d3-5fd7-4543-bf4b-6b9ba880c11b"
Jul 24 16:43:06.462234 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:06.462192    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54f4b6d0-be12-40ae-b3ac-f7acb8e715b9-metrics-certs\") pod \"network-metrics-daemon-8lwz8\" (UID: \"54f4b6d0-be12-40ae-b3ac-f7acb8e715b9\") " pod="openshift-multus/network-metrics-daemon-8lwz8"
Jul 24 16:43:06.462438 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:06.462289    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/573b3eaf-dd83-456c-b7e0-35cb5aad6884-original-pull-secret\") pod \"global-pull-secret-syncer-vzbnm\" (UID: \"573b3eaf-dd83-456c-b7e0-35cb5aad6884\") " pod="kube-system/global-pull-secret-syncer-vzbnm"
Jul 24 16:43:06.462438 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:06.462329    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 16:43:06.462438 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:06.462388    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 16:43:06.462438 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:06.462403    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54f4b6d0-be12-40ae-b3ac-f7acb8e715b9-metrics-certs podName:54f4b6d0-be12-40ae-b3ac-f7acb8e715b9 nodeName:}" failed. No retries permitted until 2026-07-24 16:43:14.462381046 +0000 UTC m=+17.104344925 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54f4b6d0-be12-40ae-b3ac-f7acb8e715b9-metrics-certs") pod "network-metrics-daemon-8lwz8" (UID: "54f4b6d0-be12-40ae-b3ac-f7acb8e715b9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 16:43:06.462438 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:06.462425    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/573b3eaf-dd83-456c-b7e0-35cb5aad6884-original-pull-secret podName:573b3eaf-dd83-456c-b7e0-35cb5aad6884 nodeName:}" failed. No retries permitted until 2026-07-24 16:43:14.462414347 +0000 UTC m=+17.104378214 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/573b3eaf-dd83-456c-b7e0-35cb5aad6884-original-pull-secret") pod "global-pull-secret-syncer-vzbnm" (UID: "573b3eaf-dd83-456c-b7e0-35cb5aad6884") : object "kube-system"/"original-pull-secret" not registered
Jul 24 16:43:06.563641 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:06.563598    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndwn5\" (UniqueName: \"kubernetes.io/projected/a12b44d3-5fd7-4543-bf4b-6b9ba880c11b-kube-api-access-ndwn5\") pod \"network-check-target-hhvqg\" (UID: \"a12b44d3-5fd7-4543-bf4b-6b9ba880c11b\") " pod="openshift-network-diagnostics/network-check-target-hhvqg"
Jul 24 16:43:06.563807 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:06.563780    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 16:43:06.563884 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:06.563813    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 16:43:06.563884 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:06.563826    2563 projected.go:196] Error preparing data for projected volume kube-api-access-ndwn5 for pod openshift-network-diagnostics/network-check-target-hhvqg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 16:43:06.563884 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:06.563882    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a12b44d3-5fd7-4543-bf4b-6b9ba880c11b-kube-api-access-ndwn5 podName:a12b44d3-5fd7-4543-bf4b-6b9ba880c11b nodeName:}" failed. No retries permitted until 2026-07-24 16:43:14.563866871 +0000 UTC m=+17.205830753 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ndwn5" (UniqueName: "kubernetes.io/projected/a12b44d3-5fd7-4543-bf4b-6b9ba880c11b-kube-api-access-ndwn5") pod "network-check-target-hhvqg" (UID: "a12b44d3-5fd7-4543-bf4b-6b9ba880c11b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 16:43:06.996523 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:06.996486    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vzbnm"
Jul 24 16:43:06.996709 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:06.996627    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vzbnm" podUID="573b3eaf-dd83-456c-b7e0-35cb5aad6884"
Jul 24 16:43:06.997126 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:06.996486    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lwz8"
Jul 24 16:43:06.997126 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:06.997028    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lwz8" podUID="54f4b6d0-be12-40ae-b3ac-f7acb8e715b9"
Jul 24 16:43:07.998269 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:07.998227    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hhvqg"
Jul 24 16:43:07.998719 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:07.998360    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hhvqg" podUID="a12b44d3-5fd7-4543-bf4b-6b9ba880c11b"
Jul 24 16:43:08.996966 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:08.996860    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lwz8"
Jul 24 16:43:08.997146 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:08.996869    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vzbnm"
Jul 24 16:43:08.997146 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:08.997037    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lwz8" podUID="54f4b6d0-be12-40ae-b3ac-f7acb8e715b9"
Jul 24 16:43:08.997146 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:08.997074    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vzbnm" podUID="573b3eaf-dd83-456c-b7e0-35cb5aad6884"
Jul 24 16:43:09.996863 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:09.996733    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hhvqg"
Jul 24 16:43:09.997384 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:09.996886    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hhvqg" podUID="a12b44d3-5fd7-4543-bf4b-6b9ba880c11b"
Jul 24 16:43:10.996415 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:10.996375    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lwz8"
Jul 24 16:43:10.996626 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:10.996426    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vzbnm"
Jul 24 16:43:10.996626 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:10.996552    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lwz8" podUID="54f4b6d0-be12-40ae-b3ac-f7acb8e715b9"
Jul 24 16:43:10.996739 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:10.996642    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vzbnm" podUID="573b3eaf-dd83-456c-b7e0-35cb5aad6884"
Jul 24 16:43:11.996186 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:11.996144    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hhvqg"
Jul 24 16:43:11.996645 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:11.996281    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hhvqg" podUID="a12b44d3-5fd7-4543-bf4b-6b9ba880c11b"
Jul 24 16:43:12.996156 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:12.996124    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lwz8"
Jul 24 16:43:12.996338 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:12.996258    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lwz8" podUID="54f4b6d0-be12-40ae-b3ac-f7acb8e715b9"
Jul 24 16:43:12.996338 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:12.996309    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vzbnm"
Jul 24 16:43:12.996695 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:12.996402    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vzbnm" podUID="573b3eaf-dd83-456c-b7e0-35cb5aad6884"
Jul 24 16:43:13.996142 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:13.996108    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hhvqg"
Jul 24 16:43:13.996291 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:13.996239    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hhvqg" podUID="a12b44d3-5fd7-4543-bf4b-6b9ba880c11b"
Jul 24 16:43:14.534848 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:14.534810    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54f4b6d0-be12-40ae-b3ac-f7acb8e715b9-metrics-certs\") pod \"network-metrics-daemon-8lwz8\" (UID: \"54f4b6d0-be12-40ae-b3ac-f7acb8e715b9\") " pod="openshift-multus/network-metrics-daemon-8lwz8"
Jul 24 16:43:14.535285 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:14.534903    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/573b3eaf-dd83-456c-b7e0-35cb5aad6884-original-pull-secret\") pod \"global-pull-secret-syncer-vzbnm\" (UID: \"573b3eaf-dd83-456c-b7e0-35cb5aad6884\") " pod="kube-system/global-pull-secret-syncer-vzbnm"
Jul 24 16:43:14.535285 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:14.534917    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 16:43:14.535285 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:14.534996    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54f4b6d0-be12-40ae-b3ac-f7acb8e715b9-metrics-certs podName:54f4b6d0-be12-40ae-b3ac-f7acb8e715b9 nodeName:}" failed. No retries permitted until 2026-07-24 16:43:30.534975552 +0000 UTC m=+33.176939424 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54f4b6d0-be12-40ae-b3ac-f7acb8e715b9-metrics-certs") pod "network-metrics-daemon-8lwz8" (UID: "54f4b6d0-be12-40ae-b3ac-f7acb8e715b9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 16:43:14.535285 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:14.535026    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 16:43:14.535285 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:14.535074    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/573b3eaf-dd83-456c-b7e0-35cb5aad6884-original-pull-secret podName:573b3eaf-dd83-456c-b7e0-35cb5aad6884 nodeName:}" failed. No retries permitted until 2026-07-24 16:43:30.535059193 +0000 UTC m=+33.177023072 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/573b3eaf-dd83-456c-b7e0-35cb5aad6884-original-pull-secret") pod "global-pull-secret-syncer-vzbnm" (UID: "573b3eaf-dd83-456c-b7e0-35cb5aad6884") : object "kube-system"/"original-pull-secret" not registered
Jul 24 16:43:14.635997 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:14.635964    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndwn5\" (UniqueName: \"kubernetes.io/projected/a12b44d3-5fd7-4543-bf4b-6b9ba880c11b-kube-api-access-ndwn5\") pod \"network-check-target-hhvqg\" (UID: \"a12b44d3-5fd7-4543-bf4b-6b9ba880c11b\") " pod="openshift-network-diagnostics/network-check-target-hhvqg"
Jul 24 16:43:14.636181 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:14.636090    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 16:43:14.636181 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:14.636107    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 16:43:14.636181 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:14.636118    2563 projected.go:196] Error preparing data for projected volume kube-api-access-ndwn5 for pod openshift-network-diagnostics/network-check-target-hhvqg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 16:43:14.636181 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:14.636166    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a12b44d3-5fd7-4543-bf4b-6b9ba880c11b-kube-api-access-ndwn5 podName:a12b44d3-5fd7-4543-bf4b-6b9ba880c11b nodeName:}" failed. No retries permitted until 2026-07-24 16:43:30.636150143 +0000 UTC m=+33.278114004 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ndwn5" (UniqueName: "kubernetes.io/projected/a12b44d3-5fd7-4543-bf4b-6b9ba880c11b-kube-api-access-ndwn5") pod "network-check-target-hhvqg" (UID: "a12b44d3-5fd7-4543-bf4b-6b9ba880c11b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 16:43:14.996233 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:14.996194    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vzbnm"
Jul 24 16:43:14.996408 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:14.996317    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vzbnm" podUID="573b3eaf-dd83-456c-b7e0-35cb5aad6884"
Jul 24 16:43:14.996408 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:14.996370    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lwz8"
Jul 24 16:43:14.996504 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:14.996482    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lwz8" podUID="54f4b6d0-be12-40ae-b3ac-f7acb8e715b9"
Jul 24 16:43:15.996807 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:15.996772    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hhvqg"
Jul 24 16:43:15.997255 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:15.996904    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hhvqg" podUID="a12b44d3-5fd7-4543-bf4b-6b9ba880c11b"
Jul 24 16:43:16.996407 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:16.996374    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lwz8"
Jul 24 16:43:16.996407 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:16.996402    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vzbnm"
Jul 24 16:43:16.996596 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:16.996522    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lwz8" podUID="54f4b6d0-be12-40ae-b3ac-f7acb8e715b9"
Jul 24 16:43:16.996668 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:16.996645    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vzbnm" podUID="573b3eaf-dd83-456c-b7e0-35cb5aad6884"
Jul 24 16:43:17.999586 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:17.999549    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hhvqg"
Jul 24 16:43:18.000042 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:17.999672    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hhvqg" podUID="a12b44d3-5fd7-4543-bf4b-6b9ba880c11b"
Jul 24 16:43:18.996870 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:18.996821    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lwz8"
Jul 24 16:43:18.996870 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:18.996856    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vzbnm"
Jul 24 16:43:18.997113 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:18.996979    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lwz8" podUID="54f4b6d0-be12-40ae-b3ac-f7acb8e715b9"
Jul 24 16:43:18.997113 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:18.997101    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vzbnm" podUID="573b3eaf-dd83-456c-b7e0-35cb5aad6884"
Jul 24 16:43:19.996262 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:19.996104    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hhvqg"
Jul 24 16:43:19.996842 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:19.996341    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hhvqg" podUID="a12b44d3-5fd7-4543-bf4b-6b9ba880c11b"
Jul 24 16:43:20.165493 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:20.165458    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vq4cx" event={"ID":"ce9a1972-6a1e-4e0d-9dab-0e25820dedf1","Type":"ContainerStarted","Data":"4231716ac97871edc99a21604555ba3c5100be8f3a23dee3a7de8f2dcafd770a"}
Jul 24 16:43:20.167020 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:20.166997    2563 generic.go:356] "Generic (PLEG): container finished" podID="5cdfb9b825abe617ad3cdd7a8d779a3b" containerID="310bff977b85ca96dce3f8eaac43a1c6c08d2f98e26ba0fd303507d25e866703" exitCode=0
Jul 24 16:43:20.167101 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:20.167064    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal" event={"ID":"5cdfb9b825abe617ad3cdd7a8d779a3b","Type":"ContainerDied","Data":"310bff977b85ca96dce3f8eaac43a1c6c08d2f98e26ba0fd303507d25e866703"}
Jul 24 16:43:20.168371 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:20.168345    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mvlt2" event={"ID":"52dcf805-dee8-447d-a8d9-221be23de2d8","Type":"ContainerStarted","Data":"85f6d9fb0bc6787da7939866501b6057a8768acb3878edf59298d1fc6b11fcbe"}
Jul 24 16:43:20.169705 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:20.169681    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfw5x" event={"ID":"3c86fd31-5d0f-4814-8cb6-825111d435d7","Type":"ContainerStarted","Data":"e2def58d641089effeab50f2e7735114fbf35ccd8b22435c1201535422fdab47"}
Jul 24 16:43:20.170929 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:20.170911    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-45v2q" event={"ID":"855016a9-43a6-4765-815e-9739917af352","Type":"ContainerStarted","Data":"1320b400893e247af609e0b38df02d2a2d714aff5843014e4440bf5a3e446895"}
Jul 24 16:43:20.172039 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:20.172020    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g5sl6" event={"ID":"f263630d-88c9-4b5d-8afb-7075a1118555","Type":"ContainerStarted","Data":"dc318f3c7296746e80960205559807610f929b5be86ef28387c759842acf576b"}
Jul 24 16:43:20.174351 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:20.174334    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk" event={"ID":"97b382f6-038a-43cc-8db6-3a08cf2d7be2","Type":"ContainerStarted","Data":"2bb25e9f6065a110fa4e390649b30ec177bc2ff6feaccf56ee751e566e755a41"}
Jul 24 16:43:20.174432 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:20.174389    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk" event={"ID":"97b382f6-038a-43cc-8db6-3a08cf2d7be2","Type":"ContainerStarted","Data":"279138b9408669edb7d79c7ad413f07050092414dfc536dfe74d33ad576003e7"}
Jul 24 16:43:20.174473 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:20.174441    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk" event={"ID":"97b382f6-038a-43cc-8db6-3a08cf2d7be2","Type":"ContainerStarted","Data":"dd20bd449ab1fb1fed6282378671920fa46644919174866f64f8381f9d7e5089"}
Jul 24 16:43:20.174507 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:20.174478    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk" event={"ID":"97b382f6-038a-43cc-8db6-3a08cf2d7be2","Type":"ContainerStarted","Data":"7a5026d877f1668ded1790eaa123ae92b4b8c1e2943a9138d2fec950474de097"}
Jul 24 16:43:20.174538 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:20.174509    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk" event={"ID":"97b382f6-038a-43cc-8db6-3a08cf2d7be2","Type":"ContainerStarted","Data":"c8ecc2ddedc4fe00da718d26a302f70b236d0c711c8355b508a51c626861c704"}
Jul 24 16:43:20.175637 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:20.175616    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dblpg" event={"ID":"e620ec4e-9703-45c1-9080-d17039c0d2e8","Type":"ContainerStarted","Data":"67834f428d6c87a7393ef4df35472ab042bffd572d2e2628ea2b562c7d2b1d5d"}
Jul 24 16:43:20.176837 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:20.176819    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-234.ec2.internal" event={"ID":"5737b582bc5c83055d2673f557e5384a","Type":"ContainerStarted","Data":"47ac29beb6035a674a00c48a7baf5148776859b7907f9854e516792bf6292516"}
Jul 24 16:43:20.178116 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:20.178098    2563 generic.go:356] "Generic (PLEG): container finished" podID="6a0daaab-0ecb-4e33-bc7c-c53caef203f2" containerID="8af88ea8130c49c3cfc24b0e6e864e67f3177798770580ee3f26774bbbb43a6b" exitCode=0
Jul 24 16:43:20.178195 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:20.178124    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4xnt" event={"ID":"6a0daaab-0ecb-4e33-bc7c-c53caef203f2","Type":"ContainerDied","Data":"8af88ea8130c49c3cfc24b0e6e864e67f3177798770580ee3f26774bbbb43a6b"}
Jul 24 16:43:20.195359 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:20.195316    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vq4cx" podStartSLOduration=1.8757451619999999 podStartE2EDuration="22.195299855s" totalImagesPullingTime="20.319554693s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 16:42:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:43:20.181438993 +0000 UTC m=+22.823402877" watchObservedRunningTime="2026-07-24 16:43:20.195299855 +0000 UTC m=+22.837263738"
Jul 24 16:43:20.196047 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:20.196013    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-g5sl6" podStartSLOduration=0.977279633 podStartE2EDuration="21.196001699s" totalImagesPullingTime="20.218722066s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 16:42:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-24 16:43:20.195891953 +0000 UTC m=+22.837855848" watchObservedRunningTime="2026-07-24 16:43:20.196001699 +0000 UTC m=+22.837965582"
Jul 24 16:43:20.235457 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:20.235401    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dblpg" podStartSLOduration=0.892972803 podStartE2EDuration="21.235385895s" totalImagesPullingTime="20.342413092s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 16:42:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:43:20.234887681 +0000 UTC m=+22.876851577" watchObservedRunningTime="2026-07-24 16:43:20.235385895 +0000 UTC m=+22.877349778"
Jul 24 16:43:20.251305 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:20.251262    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mvlt2" podStartSLOduration=1.046140489 podStartE2EDuration="21.251246807s" totalImagesPullingTime="20.205106318s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 16:42:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:43:20.25099947 +0000 UTC m=+22.892963375" watchObservedRunningTime="2026-07-24 16:43:20.251246807 +0000 UTC m=+22.893210691"
Jul 24 16:43:20.265728 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:20.265683    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-234.ec2.internal" podStartSLOduration=21.265670665 podStartE2EDuration="21.265670665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 16:42:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:43:20.265619629 +0000 UTC m=+22.907583521" watchObservedRunningTime="2026-07-24 16:43:20.265670665 +0000 UTC m=+22.907634540"
Jul 24 16:43:20.283306 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:20.283259    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-45v2q" podStartSLOduration=1.003598669 podStartE2EDuration="21.283244875s" totalImagesPullingTime="20.279646206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:42:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:43:20.282577259 +0000 UTC m=+22.924541142" watchObservedRunningTime="2026-07-24 16:43:20.283244875 +0000 UTC m=+22.925208759"
Jul 24 16:43:20.996369 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:20.996164    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lwz8"
Jul 24 16:43:20.996859 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:20.996203    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vzbnm"
Jul 24 16:43:20.996859 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:20.996463    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lwz8" podUID="54f4b6d0-be12-40ae-b3ac-f7acb8e715b9"
Jul 24 16:43:20.996859 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:20.996532    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vzbnm" podUID="573b3eaf-dd83-456c-b7e0-35cb5aad6884"
Jul 24 16:43:21.182007 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:21.181974    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gt8zs" event={"ID":"aa45a2eb-0246-4044-85cb-f09d0750d858","Type":"ContainerStarted","Data":"daaff093c531e0184014730cec9aa64e12093aa91b5613d17461b7b8efd53670"}
Jul 24 16:43:21.187004 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:21.186974    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk" event={"ID":"97b382f6-038a-43cc-8db6-3a08cf2d7be2","Type":"ContainerStarted","Data":"36978a1f3058c9045a32a4b2fd0ad0ec6308c270168ead721b24d034b6e74e32"}
Jul 24 16:43:21.196221 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:21.196171    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gt8zs" podStartSLOduration=1.940450365 podStartE2EDuration="22.196155543s" totalImagesPullingTime="20.255705178s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 16:42:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:43:21.195647111 +0000 UTC m=+23.837611014" watchObservedRunningTime="2026-07-24 16:43:21.196155543 +0000 UTC m=+23.838119427"
Jul 24 16:43:21.281771 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:21.281731    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 24 16:43:21.850055 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:21.849914    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T16:43:21.281752505Z","UUID":"3877e792-98b7-4ab7-aacc-139587c8c96c","Handler":null,"Name":"","Endpoint":""}
Jul 24 16:43:21.851619 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:21.851596    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 24 16:43:21.851738 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:21.851639    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 24 16:43:22.000695 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:22.000662    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hhvqg"
Jul 24 16:43:22.001184 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:22.000817    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hhvqg" podUID="a12b44d3-5fd7-4543-bf4b-6b9ba880c11b"
Jul 24 16:43:22.189411 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:22.189375    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal" event={"ID":"5cdfb9b825abe617ad3cdd7a8d779a3b","Type":"ContainerStarted","Data":"5c753fe449f27f2273e42d729ec22d17e2c2a65a4a668d701db74572856c9dbd"}
Jul 24 16:43:22.191036 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:22.191010    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfw5x" event={"ID":"3c86fd31-5d0f-4814-8cb6-825111d435d7","Type":"ContainerStarted","Data":"541f61fa9b0f37707c22c5d601c6a8133d720ffcdcde7888de6a765125d0b0ee"}
Jul 24 16:43:22.204217 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:22.204175    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal" podStartSLOduration=23.20415996 podStartE2EDuration="23.20415996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 16:42:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:43:22.204038405 +0000 UTC m=+24.846002289" watchObservedRunningTime="2026-07-24 16:43:22.20415996 +0000 UTC m=+24.846123845"
Jul 24 16:43:22.232555 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:22.232528    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-45v2q"
Jul 24 16:43:22.233171 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:22.233140    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-45v2q"
Jul 24 16:43:22.996352 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:22.996311    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lwz8"
Jul 24 16:43:22.996352 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:22.996338    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vzbnm"
Jul 24 16:43:22.996609 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:22.996462    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lwz8" podUID="54f4b6d0-be12-40ae-b3ac-f7acb8e715b9"
Jul 24 16:43:22.996609 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:22.996589    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vzbnm" podUID="573b3eaf-dd83-456c-b7e0-35cb5aad6884"
Jul 24 16:43:23.197632 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:23.197593    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk" event={"ID":"97b382f6-038a-43cc-8db6-3a08cf2d7be2","Type":"ContainerStarted","Data":"5655eabfe996968a2a882bbfc27f4f15541ea6c2828a42d27ea72af07396c401"}
Jul 24 16:43:23.200157 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:23.200125    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfw5x" event={"ID":"3c86fd31-5d0f-4814-8cb6-825111d435d7","Type":"ContainerStarted","Data":"28a5574ee0096b6b1b68556eda2ad630407f18743c353ee008af8c9f4df845a1"}
Jul 24 16:43:23.219677 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:23.219611    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfw5x" podStartSLOduration=1.243068565 podStartE2EDuration="24.219588093s" totalImagesPullingTime="22.976519528s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 16:42:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:43:23.219385525 +0000 UTC m=+25.861349409" watchObservedRunningTime="2026-07-24 16:43:23.219588093 +0000 UTC m=+25.861552014"
Jul 24 16:43:23.878891 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:23.878853    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-45v2q"
Jul 24 16:43:23.879608 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:23.879574    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-45v2q"
Jul 24 16:43:23.996808 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:23.996771    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hhvqg"
Jul 24 16:43:23.997002 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:23.996902    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hhvqg" podUID="a12b44d3-5fd7-4543-bf4b-6b9ba880c11b"
Jul 24 16:43:24.996026 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:24.995990    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lwz8"
Jul 24 16:43:24.996026 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:24.996019    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vzbnm"
Jul 24 16:43:24.996518 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:24.996126    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lwz8" podUID="54f4b6d0-be12-40ae-b3ac-f7acb8e715b9"
Jul 24 16:43:24.996518 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:24.996267    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vzbnm" podUID="573b3eaf-dd83-456c-b7e0-35cb5aad6884"
Jul 24 16:43:26.000029 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:25.999829    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hhvqg"
Jul 24 16:43:26.000631 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:26.000099    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hhvqg" podUID="a12b44d3-5fd7-4543-bf4b-6b9ba880c11b"
Jul 24 16:43:26.207893 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:26.207811    2563 generic.go:356] "Generic (PLEG): container finished" podID="6a0daaab-0ecb-4e33-bc7c-c53caef203f2" containerID="7cf571014a9d24bfa1d44401781aeca00e1869d43d8ca1456184359549d30efa" exitCode=0
Jul 24 16:43:26.208071 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:26.207893    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4xnt" event={"ID":"6a0daaab-0ecb-4e33-bc7c-c53caef203f2","Type":"ContainerDied","Data":"7cf571014a9d24bfa1d44401781aeca00e1869d43d8ca1456184359549d30efa"}
Jul 24 16:43:26.212401 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:26.212377    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk" event={"ID":"97b382f6-038a-43cc-8db6-3a08cf2d7be2","Type":"ContainerStarted","Data":"ec0c346f4544ddb1bf11ccb2ea0903e512358540936d808375e7846ef20e5d7c"}
Jul 24 16:43:26.212677 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:26.212639    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:43:26.212793 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:26.212781    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:43:26.212861 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:26.212827    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:43:26.228101 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:26.228078    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:43:26.228228 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:26.228175    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:43:26.255589 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:26.255546    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk" podStartSLOduration=6.942910951 podStartE2EDuration="27.255529872s" totalImagesPullingTime="20.312618921s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 16:42:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:43:26.25518905 +0000 UTC m=+28.897152934" watchObservedRunningTime="2026-07-24 16:43:26.255529872 +0000 UTC m=+28.897493747"
Jul 24 16:43:26.996861 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:26.996824    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lwz8"
Jul 24 16:43:26.997166 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:26.996824    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vzbnm"
Jul 24 16:43:26.997166 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:26.997006    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lwz8" podUID="54f4b6d0-be12-40ae-b3ac-f7acb8e715b9"
Jul 24 16:43:26.997277 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:26.997170    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vzbnm" podUID="573b3eaf-dd83-456c-b7e0-35cb5aad6884"
Jul 24 16:43:27.997344 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:27.997159    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hhvqg"
Jul 24 16:43:27.997842 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:27.997406    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hhvqg" podUID="a12b44d3-5fd7-4543-bf4b-6b9ba880c11b"
Jul 24 16:43:28.024891 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:28.024862    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vzbnm"]
Jul 24 16:43:28.025078 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:28.024981    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vzbnm"
Jul 24 16:43:28.025078 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:28.025066    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vzbnm" podUID="573b3eaf-dd83-456c-b7e0-35cb5aad6884"
Jul 24 16:43:28.027558 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:28.027531    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hhvqg"]
Jul 24 16:43:28.039771 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:28.039748    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8lwz8"]
Jul 24 16:43:28.040025 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:28.039857    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lwz8"
Jul 24 16:43:28.040025 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:28.039963    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lwz8" podUID="54f4b6d0-be12-40ae-b3ac-f7acb8e715b9"
Jul 24 16:43:28.218828 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:28.218794    2563 generic.go:356] "Generic (PLEG): container finished" podID="6a0daaab-0ecb-4e33-bc7c-c53caef203f2" containerID="98fe08a05e0d60f4287035156c7f019994d8bf74a543139f740a342719746876" exitCode=0
Jul 24 16:43:28.219033 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:28.218882    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4xnt" event={"ID":"6a0daaab-0ecb-4e33-bc7c-c53caef203f2","Type":"ContainerDied","Data":"98fe08a05e0d60f4287035156c7f019994d8bf74a543139f740a342719746876"}
Jul 24 16:43:28.219033 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:28.218924    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hhvqg"
Jul 24 16:43:28.219524 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:28.219493    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hhvqg" podUID="a12b44d3-5fd7-4543-bf4b-6b9ba880c11b"
Jul 24 16:43:29.995998 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:29.995964    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hhvqg"
Jul 24 16:43:29.996431 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:29.995970    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vzbnm"
Jul 24 16:43:29.996431 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:29.996100    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hhvqg" podUID="a12b44d3-5fd7-4543-bf4b-6b9ba880c11b"
Jul 24 16:43:29.996431 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:29.995976    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lwz8"
Jul 24 16:43:29.996649 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:29.996620    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lwz8" podUID="54f4b6d0-be12-40ae-b3ac-f7acb8e715b9"
Jul 24 16:43:29.997070 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:29.997043    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vzbnm" podUID="573b3eaf-dd83-456c-b7e0-35cb5aad6884"
Jul 24 16:43:30.227566 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:30.227535    2563 generic.go:356] "Generic (PLEG): container finished" podID="6a0daaab-0ecb-4e33-bc7c-c53caef203f2" containerID="2a9560f82040a10f4ef0ff2716534b22034f44d6addaa8db5c754035c5ee5e1c" exitCode=0
Jul 24 16:43:30.227761 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:30.227594    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4xnt" event={"ID":"6a0daaab-0ecb-4e33-bc7c-c53caef203f2","Type":"ContainerDied","Data":"2a9560f82040a10f4ef0ff2716534b22034f44d6addaa8db5c754035c5ee5e1c"}
Jul 24 16:43:30.546815 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:30.546700    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/573b3eaf-dd83-456c-b7e0-35cb5aad6884-original-pull-secret\") pod \"global-pull-secret-syncer-vzbnm\" (UID: \"573b3eaf-dd83-456c-b7e0-35cb5aad6884\") " pod="kube-system/global-pull-secret-syncer-vzbnm"
Jul 24 16:43:30.546815 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:30.546768    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54f4b6d0-be12-40ae-b3ac-f7acb8e715b9-metrics-certs\") pod \"network-metrics-daemon-8lwz8\" (UID: \"54f4b6d0-be12-40ae-b3ac-f7acb8e715b9\") " pod="openshift-multus/network-metrics-daemon-8lwz8"
Jul 24 16:43:30.547049 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:30.546846    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 16:43:30.547049 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:30.546872    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 16:43:30.547049 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:30.546908    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/573b3eaf-dd83-456c-b7e0-35cb5aad6884-original-pull-secret podName:573b3eaf-dd83-456c-b7e0-35cb5aad6884 nodeName:}" failed. No retries permitted until 2026-07-24 16:44:02.546893463 +0000 UTC m=+65.188857326 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/573b3eaf-dd83-456c-b7e0-35cb5aad6884-original-pull-secret") pod "global-pull-secret-syncer-vzbnm" (UID: "573b3eaf-dd83-456c-b7e0-35cb5aad6884") : object "kube-system"/"original-pull-secret" not registered
Jul 24 16:43:30.547049 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:30.546930    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54f4b6d0-be12-40ae-b3ac-f7acb8e715b9-metrics-certs podName:54f4b6d0-be12-40ae-b3ac-f7acb8e715b9 nodeName:}" failed. No retries permitted until 2026-07-24 16:44:02.54692276 +0000 UTC m=+65.188886622 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54f4b6d0-be12-40ae-b3ac-f7acb8e715b9-metrics-certs") pod "network-metrics-daemon-8lwz8" (UID: "54f4b6d0-be12-40ae-b3ac-f7acb8e715b9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 16:43:30.647129 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:30.647081    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndwn5\" (UniqueName: \"kubernetes.io/projected/a12b44d3-5fd7-4543-bf4b-6b9ba880c11b-kube-api-access-ndwn5\") pod \"network-check-target-hhvqg\" (UID: \"a12b44d3-5fd7-4543-bf4b-6b9ba880c11b\") " pod="openshift-network-diagnostics/network-check-target-hhvqg"
Jul 24 16:43:30.647284 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:30.647216    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 16:43:30.647284 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:30.647232    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 16:43:30.647284 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:30.647241    2563 projected.go:196] Error preparing data for projected volume kube-api-access-ndwn5 for pod openshift-network-diagnostics/network-check-target-hhvqg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 16:43:30.647403 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:30.647289    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a12b44d3-5fd7-4543-bf4b-6b9ba880c11b-kube-api-access-ndwn5 podName:a12b44d3-5fd7-4543-bf4b-6b9ba880c11b nodeName:}" failed. No retries permitted until 2026-07-24 16:44:02.64727755 +0000 UTC m=+65.289241412 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ndwn5" (UniqueName: "kubernetes.io/projected/a12b44d3-5fd7-4543-bf4b-6b9ba880c11b-kube-api-access-ndwn5") pod "network-check-target-hhvqg" (UID: "a12b44d3-5fd7-4543-bf4b-6b9ba880c11b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 16:43:31.996164 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:31.996134    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vzbnm"
Jul 24 16:43:31.996164 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:31.996144    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lwz8"
Jul 24 16:43:31.996570 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:31.996253    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vzbnm" podUID="573b3eaf-dd83-456c-b7e0-35cb5aad6884"
Jul 24 16:43:31.996570 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:31.996318    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8lwz8" podUID="54f4b6d0-be12-40ae-b3ac-f7acb8e715b9"
Jul 24 16:43:31.996570 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:31.996363    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hhvqg"
Jul 24 16:43:31.996570 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:31.996466    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hhvqg" podUID="a12b44d3-5fd7-4543-bf4b-6b9ba880c11b"
Jul 24 16:43:32.191408 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.191332    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeReady"
Jul 24 16:43:32.191565 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.191470    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 24 16:43:32.224397 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.224358    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-mt8vp"]
Jul 24 16:43:32.227870 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.227845    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-c8f78f7cf-2sdpb"]
Jul 24 16:43:32.228026 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.228010    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-mt8vp"
Jul 24 16:43:32.230725 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.230482    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 24 16:43:32.230725 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.230491    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 24 16:43:32.230725 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.230599    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-rdbwx\""
Jul 24 16:43:32.232217 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.232199    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:43:32.234144 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.234125    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 24 16:43:32.234656 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.234477    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 24 16:43:32.234656 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.234518    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4mtk4\""
Jul 24 16:43:32.234656 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.234476    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 24 16:43:32.238267 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.237561    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vrscm"]
Jul 24 16:43:32.242215 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.241234    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8zdrn"]
Jul 24 16:43:32.243485 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.243461    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vrscm"
Jul 24 16:43:32.244093 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.244073    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 24 16:43:32.245969 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.245598    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6r7g7\""
Jul 24 16:43:32.248816 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.247401    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 24 16:43:32.248816 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.247660    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 24 16:43:32.248816 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.247793    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-mt8vp"]
Jul 24 16:43:32.248816 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.247826    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c8f78f7cf-2sdpb"]
Jul 24 16:43:32.248816 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.247926    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8zdrn"
Jul 24 16:43:32.250154 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.250127    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 24 16:43:32.250356 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.250316    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nk7wj\""
Jul 24 16:43:32.250447 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.250390    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 24 16:43:32.250447 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.250435    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 24 16:43:32.250880 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.250862    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8zdrn"]
Jul 24 16:43:32.258769 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.258747    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vrscm"]
Jul 24 16:43:32.361597 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.361567    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm9jh\" (UniqueName: \"kubernetes.io/projected/17291b08-516e-48fe-9b2a-e8db21ed7c6c-kube-api-access-tm9jh\") pod \"ingress-canary-8zdrn\" (UID: \"17291b08-516e-48fe-9b2a-e8db21ed7c6c\") " pod="openshift-ingress-canary/ingress-canary-8zdrn"
Jul 24 16:43:32.361597 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.361609    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-image-registry-private-configuration\") pod \"image-registry-c8f78f7cf-2sdpb\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") " pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:43:32.362076 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.361655    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8jjw\" (UniqueName: \"kubernetes.io/projected/708abf59-5c2a-46e2-8506-386e91731078-kube-api-access-t8jjw\") pod \"dns-default-vrscm\" (UID: \"708abf59-5c2a-46e2-8506-386e91731078\") " pod="openshift-dns/dns-default-vrscm"
Jul 24 16:43:32.362076 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.361685    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7960f9f4-815e-4b60-bd8c-2a0654a61ae6-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-mt8vp\" (UID: \"7960f9f4-815e-4b60-bd8c-2a0654a61ae6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-mt8vp"
Jul 24 16:43:32.362076 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.361721    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-ca-trust-extracted\") pod \"image-registry-c8f78f7cf-2sdpb\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") " pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:43:32.362076 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.361738    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-installation-pull-secrets\") pod \"image-registry-c8f78f7cf-2sdpb\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") " pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:43:32.362076 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.361761    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p29fm\" (UniqueName: \"kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-kube-api-access-p29fm\") pod \"image-registry-c8f78f7cf-2sdpb\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") " pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:43:32.362076 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.361793    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/708abf59-5c2a-46e2-8506-386e91731078-tmp-dir\") pod \"dns-default-vrscm\" (UID: \"708abf59-5c2a-46e2-8506-386e91731078\") " pod="openshift-dns/dns-default-vrscm"
Jul 24 16:43:32.362076 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.361811    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-registry-certificates\") pod \"image-registry-c8f78f7cf-2sdpb\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") " pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:43:32.362076 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.361902    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/708abf59-5c2a-46e2-8506-386e91731078-config-volume\") pod \"dns-default-vrscm\" (UID: \"708abf59-5c2a-46e2-8506-386e91731078\") " pod="openshift-dns/dns-default-vrscm"
Jul 24 16:43:32.362076 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.361968    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/708abf59-5c2a-46e2-8506-386e91731078-metrics-tls\") pod \"dns-default-vrscm\" (UID: \"708abf59-5c2a-46e2-8506-386e91731078\") " pod="openshift-dns/dns-default-vrscm"
Jul 24 16:43:32.362076 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.362007    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7960f9f4-815e-4b60-bd8c-2a0654a61ae6-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-mt8vp\" (UID: \"7960f9f4-815e-4b60-bd8c-2a0654a61ae6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-mt8vp"
Jul 24 16:43:32.362076 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.362032    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-registry-tls\") pod \"image-registry-c8f78f7cf-2sdpb\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") " pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:43:32.362076 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.362049    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-bound-sa-token\") pod \"image-registry-c8f78f7cf-2sdpb\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") " pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:43:32.362076 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.362067    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/17291b08-516e-48fe-9b2a-e8db21ed7c6c-cert\") pod \"ingress-canary-8zdrn\" (UID: \"17291b08-516e-48fe-9b2a-e8db21ed7c6c\") " pod="openshift-ingress-canary/ingress-canary-8zdrn"
Jul 24 16:43:32.362777 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.362137    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-trusted-ca\") pod \"image-registry-c8f78f7cf-2sdpb\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") " pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:43:32.380051 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.380008    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:43:32.462958 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.462855    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7960f9f4-815e-4b60-bd8c-2a0654a61ae6-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-mt8vp\" (UID: \"7960f9f4-815e-4b60-bd8c-2a0654a61ae6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-mt8vp"
Jul 24 16:43:32.462958 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.462897    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-registry-tls\") pod \"image-registry-c8f78f7cf-2sdpb\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") " pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:43:32.462958 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.462924    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-bound-sa-token\") pod \"image-registry-c8f78f7cf-2sdpb\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") " pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:43:32.463211 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:32.463054    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 16:43:32.463211 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:32.463076    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c8f78f7cf-2sdpb: secret "image-registry-tls" not found
Jul 24 16:43:32.463211 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.463092    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/17291b08-516e-48fe-9b2a-e8db21ed7c6c-cert\") pod \"ingress-canary-8zdrn\" (UID: \"17291b08-516e-48fe-9b2a-e8db21ed7c6c\") " pod="openshift-ingress-canary/ingress-canary-8zdrn"
Jul 24 16:43:32.463211 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:32.463138    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-registry-tls podName:6e7f3a6f-1da1-4a07-b736-512dfb2afd6a nodeName:}" failed. No retries permitted until 2026-07-24 16:43:32.963115002 +0000 UTC m=+35.605078885 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-registry-tls") pod "image-registry-c8f78f7cf-2sdpb" (UID: "6e7f3a6f-1da1-4a07-b736-512dfb2afd6a") : secret "image-registry-tls" not found
Jul 24 16:43:32.463211 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.463188    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-trusted-ca\") pod \"image-registry-c8f78f7cf-2sdpb\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") " pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:43:32.463401 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.463232    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm9jh\" (UniqueName: \"kubernetes.io/projected/17291b08-516e-48fe-9b2a-e8db21ed7c6c-kube-api-access-tm9jh\") pod \"ingress-canary-8zdrn\" (UID: \"17291b08-516e-48fe-9b2a-e8db21ed7c6c\") " pod="openshift-ingress-canary/ingress-canary-8zdrn"
Jul 24 16:43:32.463401 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.463267    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-image-registry-private-configuration\") pod \"image-registry-c8f78f7cf-2sdpb\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") " pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:43:32.463401 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.463321    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8jjw\" (UniqueName: \"kubernetes.io/projected/708abf59-5c2a-46e2-8506-386e91731078-kube-api-access-t8jjw\") pod \"dns-default-vrscm\" (UID: \"708abf59-5c2a-46e2-8506-386e91731078\") " pod="openshift-dns/dns-default-vrscm"
Jul 24 16:43:32.463401 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.463347    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7960f9f4-815e-4b60-bd8c-2a0654a61ae6-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-mt8vp\" (UID: \"7960f9f4-815e-4b60-bd8c-2a0654a61ae6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-mt8vp"
Jul 24 16:43:32.463401 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.463376    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-ca-trust-extracted\") pod \"image-registry-c8f78f7cf-2sdpb\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") " pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:43:32.463556 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.463402    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-installation-pull-secrets\") pod \"image-registry-c8f78f7cf-2sdpb\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") " pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:43:32.463556 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.463427    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p29fm\" (UniqueName: \"kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-kube-api-access-p29fm\") pod \"image-registry-c8f78f7cf-2sdpb\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") " pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:43:32.463556 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.463463    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/708abf59-5c2a-46e2-8506-386e91731078-tmp-dir\") pod \"dns-default-vrscm\" (UID: \"708abf59-5c2a-46e2-8506-386e91731078\") " pod="openshift-dns/dns-default-vrscm"
Jul 24 16:43:32.463556 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.463483    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-registry-certificates\") pod \"image-registry-c8f78f7cf-2sdpb\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") " pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:43:32.463556 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.463541    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/708abf59-5c2a-46e2-8506-386e91731078-config-volume\") pod \"dns-default-vrscm\" (UID: \"708abf59-5c2a-46e2-8506-386e91731078\") " pod="openshift-dns/dns-default-vrscm"
Jul 24 16:43:32.463703 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.463571    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/708abf59-5c2a-46e2-8506-386e91731078-metrics-tls\") pod \"dns-default-vrscm\" (UID: \"708abf59-5c2a-46e2-8506-386e91731078\") " pod="openshift-dns/dns-default-vrscm"
Jul 24 16:43:32.463703 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:32.463681    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 16:43:32.463785 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:32.463727    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/708abf59-5c2a-46e2-8506-386e91731078-metrics-tls podName:708abf59-5c2a-46e2-8506-386e91731078 nodeName:}" failed. No retries permitted until 2026-07-24 16:43:32.963711276 +0000 UTC m=+35.605675141 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/708abf59-5c2a-46e2-8506-386e91731078-metrics-tls") pod "dns-default-vrscm" (UID: "708abf59-5c2a-46e2-8506-386e91731078") : secret "dns-default-metrics-tls" not found
Jul 24 16:43:32.463785 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.463763    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7960f9f4-815e-4b60-bd8c-2a0654a61ae6-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-mt8vp\" (UID: \"7960f9f4-815e-4b60-bd8c-2a0654a61ae6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-mt8vp"
Jul 24 16:43:32.464504 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.464479    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/708abf59-5c2a-46e2-8506-386e91731078-config-volume\") pod \"dns-default-vrscm\" (UID: \"708abf59-5c2a-46e2-8506-386e91731078\") " pod="openshift-dns/dns-default-vrscm"
Jul 24 16:43:32.464617 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.464482    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/708abf59-5c2a-46e2-8506-386e91731078-tmp-dir\") pod \"dns-default-vrscm\" (UID: \"708abf59-5c2a-46e2-8506-386e91731078\") " pod="openshift-dns/dns-default-vrscm"
Jul 24 16:43:32.464617 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:32.464547    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 16:43:32.464617 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.464546    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-ca-trust-extracted\") pod \"image-registry-c8f78f7cf-2sdpb\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") " pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:43:32.464617 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:32.464596    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7960f9f4-815e-4b60-bd8c-2a0654a61ae6-networking-console-plugin-cert podName:7960f9f4-815e-4b60-bd8c-2a0654a61ae6 nodeName:}" failed. No retries permitted until 2026-07-24 16:43:32.964579513 +0000 UTC m=+35.606543388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7960f9f4-815e-4b60-bd8c-2a0654a61ae6-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-mt8vp" (UID: "7960f9f4-815e-4b60-bd8c-2a0654a61ae6") : secret "networking-console-plugin-cert" not found
Jul 24 16:43:32.464915 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.464894    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-trusted-ca\") pod \"image-registry-c8f78f7cf-2sdpb\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") " pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:43:32.465298 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.465079    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-registry-certificates\") pod \"image-registry-c8f78f7cf-2sdpb\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") " pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:43:32.468767 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.468639    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-installation-pull-secrets\") pod \"image-registry-c8f78f7cf-2sdpb\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") " pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:43:32.468848 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.468661    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-image-registry-private-configuration\") pod \"image-registry-c8f78f7cf-2sdpb\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") " pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:43:32.468848 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.468711    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/17291b08-516e-48fe-9b2a-e8db21ed7c6c-cert\") pod \"ingress-canary-8zdrn\" (UID: \"17291b08-516e-48fe-9b2a-e8db21ed7c6c\") " pod="openshift-ingress-canary/ingress-canary-8zdrn"
Jul 24 16:43:32.472227 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.472181    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8jjw\" (UniqueName: \"kubernetes.io/projected/708abf59-5c2a-46e2-8506-386e91731078-kube-api-access-t8jjw\") pod \"dns-default-vrscm\" (UID: \"708abf59-5c2a-46e2-8506-386e91731078\") " pod="openshift-dns/dns-default-vrscm"
Jul 24 16:43:32.472327 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.472302    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-bound-sa-token\") pod \"image-registry-c8f78f7cf-2sdpb\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") " pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:43:32.472654 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.472633    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm9jh\" (UniqueName: \"kubernetes.io/projected/17291b08-516e-48fe-9b2a-e8db21ed7c6c-kube-api-access-tm9jh\") pod \"ingress-canary-8zdrn\" (UID: \"17291b08-516e-48fe-9b2a-e8db21ed7c6c\") " pod="openshift-ingress-canary/ingress-canary-8zdrn"
Jul 24 16:43:32.472743 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.472723    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p29fm\" (UniqueName: \"kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-kube-api-access-p29fm\") pod \"image-registry-c8f78f7cf-2sdpb\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") " pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:43:32.501668 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.501590    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:43:32.566924 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.566889    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8zdrn"
Jul 24 16:43:32.723336 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.723257    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8zdrn"]
Jul 24 16:43:32.727244 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:43:32.727215    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17291b08_516e_48fe_9b2a_e8db21ed7c6c.slice/crio-95506c98943eec28a603350a140f3a59bd469f0bcaf4bcc421002374f9d014a2 WatchSource:0}: Error finding container 95506c98943eec28a603350a140f3a59bd469f0bcaf4bcc421002374f9d014a2: Status 404 returned error can't find the container with id 95506c98943eec28a603350a140f3a59bd469f0bcaf4bcc421002374f9d014a2
Jul 24 16:43:32.968509 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.968473    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/708abf59-5c2a-46e2-8506-386e91731078-metrics-tls\") pod \"dns-default-vrscm\" (UID: \"708abf59-5c2a-46e2-8506-386e91731078\") " pod="openshift-dns/dns-default-vrscm"
Jul 24 16:43:32.968682 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.968530    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-registry-tls\") pod \"image-registry-c8f78f7cf-2sdpb\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") " pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:43:32.968682 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:32.968632    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 16:43:32.968682 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:32.968643    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c8f78f7cf-2sdpb: secret "image-registry-tls" not found
Jul 24 16:43:32.968682 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:32.968654    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7960f9f4-815e-4b60-bd8c-2a0654a61ae6-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-mt8vp\" (UID: \"7960f9f4-815e-4b60-bd8c-2a0654a61ae6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-mt8vp"
Jul 24 16:43:32.968837 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:32.968700    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-registry-tls podName:6e7f3a6f-1da1-4a07-b736-512dfb2afd6a nodeName:}" failed. No retries permitted until 2026-07-24 16:43:33.968686273 +0000 UTC m=+36.610650164 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-registry-tls") pod "image-registry-c8f78f7cf-2sdpb" (UID: "6e7f3a6f-1da1-4a07-b736-512dfb2afd6a") : secret "image-registry-tls" not found
Jul 24 16:43:32.968837 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:32.968739    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 16:43:32.968837 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:32.968746    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 16:43:32.968837 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:32.968785    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7960f9f4-815e-4b60-bd8c-2a0654a61ae6-networking-console-plugin-cert podName:7960f9f4-815e-4b60-bd8c-2a0654a61ae6 nodeName:}" failed. No retries permitted until 2026-07-24 16:43:33.968773937 +0000 UTC m=+36.610737833 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7960f9f4-815e-4b60-bd8c-2a0654a61ae6-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-mt8vp" (UID: "7960f9f4-815e-4b60-bd8c-2a0654a61ae6") : secret "networking-console-plugin-cert" not found
Jul 24 16:43:32.968837 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:32.968797    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/708abf59-5c2a-46e2-8506-386e91731078-metrics-tls podName:708abf59-5c2a-46e2-8506-386e91731078 nodeName:}" failed. No retries permitted until 2026-07-24 16:43:33.968791078 +0000 UTC m=+36.610754939 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/708abf59-5c2a-46e2-8506-386e91731078-metrics-tls") pod "dns-default-vrscm" (UID: "708abf59-5c2a-46e2-8506-386e91731078") : secret "dns-default-metrics-tls" not found
Jul 24 16:43:33.236452 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:33.236417    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8zdrn" event={"ID":"17291b08-516e-48fe-9b2a-e8db21ed7c6c","Type":"ContainerStarted","Data":"95506c98943eec28a603350a140f3a59bd469f0bcaf4bcc421002374f9d014a2"}
Jul 24 16:43:33.978822 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:33.978784    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/708abf59-5c2a-46e2-8506-386e91731078-metrics-tls\") pod \"dns-default-vrscm\" (UID: \"708abf59-5c2a-46e2-8506-386e91731078\") " pod="openshift-dns/dns-default-vrscm"
Jul 24 16:43:33.978822 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:33.978825    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-registry-tls\") pod \"image-registry-c8f78f7cf-2sdpb\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") " pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:43:33.979107 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:33.978899    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7960f9f4-815e-4b60-bd8c-2a0654a61ae6-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-mt8vp\" (UID: \"7960f9f4-815e-4b60-bd8c-2a0654a61ae6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-mt8vp"
Jul 24 16:43:33.979107 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:33.978974    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 16:43:33.979107 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:33.979018    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 16:43:33.979107 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:33.979023    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 16:43:33.979107 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:33.979064    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c8f78f7cf-2sdpb: secret "image-registry-tls" not found
Jul 24 16:43:33.979107 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:33.979056    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/708abf59-5c2a-46e2-8506-386e91731078-metrics-tls podName:708abf59-5c2a-46e2-8506-386e91731078 nodeName:}" failed. No retries permitted until 2026-07-24 16:43:35.979032804 +0000 UTC m=+38.620996682 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/708abf59-5c2a-46e2-8506-386e91731078-metrics-tls") pod "dns-default-vrscm" (UID: "708abf59-5c2a-46e2-8506-386e91731078") : secret "dns-default-metrics-tls" not found
Jul 24 16:43:33.979107 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:33.979102    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7960f9f4-815e-4b60-bd8c-2a0654a61ae6-networking-console-plugin-cert podName:7960f9f4-815e-4b60-bd8c-2a0654a61ae6 nodeName:}" failed. No retries permitted until 2026-07-24 16:43:35.979086099 +0000 UTC m=+38.621049964 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7960f9f4-815e-4b60-bd8c-2a0654a61ae6-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-mt8vp" (UID: "7960f9f4-815e-4b60-bd8c-2a0654a61ae6") : secret "networking-console-plugin-cert" not found
Jul 24 16:43:33.979418 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:33.979126    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-registry-tls podName:6e7f3a6f-1da1-4a07-b736-512dfb2afd6a nodeName:}" failed. No retries permitted until 2026-07-24 16:43:35.97911266 +0000 UTC m=+38.621076522 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-registry-tls") pod "image-registry-c8f78f7cf-2sdpb" (UID: "6e7f3a6f-1da1-4a07-b736-512dfb2afd6a") : secret "image-registry-tls" not found
Jul 24 16:43:33.996228 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:33.996198    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vzbnm"
Jul 24 16:43:33.996401 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:33.996198    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hhvqg"
Jul 24 16:43:33.996401 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:33.996206    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lwz8"
Jul 24 16:43:33.998535 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:33.998510    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 16:43:33.998672 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:33.998520    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 16:43:33.999516 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:33.999478    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 16:43:33.999626 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:33.999547    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nn52b\""
Jul 24 16:43:33.999626 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:33.999605    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 16:43:33.999790 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:33.999765    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6gtf2\""
Jul 24 16:43:34.728711 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:34.728657    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:43:34.790605 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:34.790569    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:43:35.428905 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:35.428869    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:43:35.828523 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:35.828452    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:43:35.927584 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:35.927555    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:43:35.998253 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:35.998215    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/708abf59-5c2a-46e2-8506-386e91731078-metrics-tls\") pod \"dns-default-vrscm\" (UID: \"708abf59-5c2a-46e2-8506-386e91731078\") " pod="openshift-dns/dns-default-vrscm"
Jul 24 16:43:35.998253 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:35.998262    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-registry-tls\") pod \"image-registry-c8f78f7cf-2sdpb\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") " pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:43:35.998524 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:35.998351    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 16:43:35.998524 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:35.998422    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 16:43:35.998524 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:35.998348    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7960f9f4-815e-4b60-bd8c-2a0654a61ae6-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-mt8vp\" (UID: \"7960f9f4-815e-4b60-bd8c-2a0654a61ae6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-mt8vp"
Jul 24 16:43:35.998524 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:35.998438    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c8f78f7cf-2sdpb: secret "image-registry-tls" not found
Jul 24 16:43:35.998524 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:35.998448    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 16:43:35.998524 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:35.998425    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/708abf59-5c2a-46e2-8506-386e91731078-metrics-tls podName:708abf59-5c2a-46e2-8506-386e91731078 nodeName:}" failed. No retries permitted until 2026-07-24 16:43:39.998408662 +0000 UTC m=+42.640372524 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/708abf59-5c2a-46e2-8506-386e91731078-metrics-tls") pod "dns-default-vrscm" (UID: "708abf59-5c2a-46e2-8506-386e91731078") : secret "dns-default-metrics-tls" not found
Jul 24 16:43:35.998524 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:35.998502    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-registry-tls podName:6e7f3a6f-1da1-4a07-b736-512dfb2afd6a nodeName:}" failed. No retries permitted until 2026-07-24 16:43:39.99848797 +0000 UTC m=+42.640451834 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-registry-tls") pod "image-registry-c8f78f7cf-2sdpb" (UID: "6e7f3a6f-1da1-4a07-b736-512dfb2afd6a") : secret "image-registry-tls" not found
Jul 24 16:43:35.998524 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:35.998522    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7960f9f4-815e-4b60-bd8c-2a0654a61ae6-networking-console-plugin-cert podName:7960f9f4-815e-4b60-bd8c-2a0654a61ae6 nodeName:}" failed. No retries permitted until 2026-07-24 16:43:39.998510916 +0000 UTC m=+42.640474778 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7960f9f4-815e-4b60-bd8c-2a0654a61ae6-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-mt8vp" (UID: "7960f9f4-815e-4b60-bd8c-2a0654a61ae6") : secret "networking-console-plugin-cert" not found
Jul 24 16:43:36.028003 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:36.027969    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:43:36.248532 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:36.248213    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4xnt" event={"ID":"6a0daaab-0ecb-4e33-bc7c-c53caef203f2","Type":"ContainerStarted","Data":"e9a6133dd993a37175d7309d743df55ef5ca8f696ba9e6f3a9288b6910c02684"}
Jul 24 16:43:36.250079 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:36.250054    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8zdrn" event={"ID":"17291b08-516e-48fe-9b2a-e8db21ed7c6c","Type":"ContainerStarted","Data":"f5afb55063fb2fe3e5acd858a8049196e11d7b52a03eb35f1ad3475b424a7c15"}
Jul 24 16:43:36.889102 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:36.889064    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:43:36.944220 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:36.944164    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:43:37.255331 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:37.255248    2563 generic.go:356] "Generic (PLEG): container finished" podID="6a0daaab-0ecb-4e33-bc7c-c53caef203f2" containerID="e9a6133dd993a37175d7309d743df55ef5ca8f696ba9e6f3a9288b6910c02684" exitCode=0
Jul 24 16:43:37.255473 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:37.255352    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4xnt" event={"ID":"6a0daaab-0ecb-4e33-bc7c-c53caef203f2","Type":"ContainerDied","Data":"e9a6133dd993a37175d7309d743df55ef5ca8f696ba9e6f3a9288b6910c02684"}
Jul 24 16:43:37.280950 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:37.280890    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8zdrn" podStartSLOduration=1.953480271 podStartE2EDuration="5.280875216s" totalImagesPullingTime="3.327394945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:43:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:43:36.284692776 +0000 UTC m=+38.926656679" watchObservedRunningTime="2026-07-24 16:43:37.280875216 +0000 UTC m=+39.922839099"
Jul 24 16:43:38.057597 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:38.057563    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:43:38.114761 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:38.114726    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:43:38.260186 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:38.260159    2563 generic.go:356] "Generic (PLEG): container finished" podID="6a0daaab-0ecb-4e33-bc7c-c53caef203f2" containerID="b78fd133fb076016990dc5e28b9fbb19ace5f9eef5cee3cd26cfbd4c9e49c1d6" exitCode=0
Jul 24 16:43:38.260311 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:38.260195    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4xnt" event={"ID":"6a0daaab-0ecb-4e33-bc7c-c53caef203f2","Type":"ContainerDied","Data":"b78fd133fb076016990dc5e28b9fbb19ace5f9eef5cee3cd26cfbd4c9e49c1d6"}
Jul 24 16:43:38.792569 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:38.792536    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:43:38.847748 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:38.847714    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:43:39.265831 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:39.265797    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4xnt" event={"ID":"6a0daaab-0ecb-4e33-bc7c-c53caef203f2","Type":"ContainerStarted","Data":"7f46fa0d37518bddaa71407c81632481a695cf47f9cba4c0f4272a48667777d1"}
Jul 24 16:43:39.289279 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:39.289239    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-t4xnt" podStartSLOduration=7.091959797 podStartE2EDuration="41.289225556s" totalImagesPullingTime="34.197265759s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 16:42:58 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:43:39.287966726 +0000 UTC m=+41.929930604" watchObservedRunningTime="2026-07-24 16:43:39.289225556 +0000 UTC m=+41.931189440"
Jul 24 16:43:39.410467 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:39.410440    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:43:39.466383 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:39.466347    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:43:40.030967 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:40.030925    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7960f9f4-815e-4b60-bd8c-2a0654a61ae6-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-mt8vp\" (UID: \"7960f9f4-815e-4b60-bd8c-2a0654a61ae6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-mt8vp"
Jul 24 16:43:40.031132 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:40.030990    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/708abf59-5c2a-46e2-8506-386e91731078-metrics-tls\") pod \"dns-default-vrscm\" (UID: \"708abf59-5c2a-46e2-8506-386e91731078\") " pod="openshift-dns/dns-default-vrscm"
Jul 24 16:43:40.031132 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:40.031012    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-registry-tls\") pod \"image-registry-c8f78f7cf-2sdpb\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") " pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:43:40.031132 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:40.031084    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 16:43:40.031132 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:40.031101    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 16:43:40.031132 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:40.031110    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c8f78f7cf-2sdpb: secret "image-registry-tls" not found
Jul 24 16:43:40.031132 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:40.031125    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 16:43:40.031418 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:40.031146    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-registry-tls podName:6e7f3a6f-1da1-4a07-b736-512dfb2afd6a nodeName:}" failed. No retries permitted until 2026-07-24 16:43:48.031135238 +0000 UTC m=+50.673099100 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-registry-tls") pod "image-registry-c8f78f7cf-2sdpb" (UID: "6e7f3a6f-1da1-4a07-b736-512dfb2afd6a") : secret "image-registry-tls" not found
Jul 24 16:43:40.031418 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:40.031159    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7960f9f4-815e-4b60-bd8c-2a0654a61ae6-networking-console-plugin-cert podName:7960f9f4-815e-4b60-bd8c-2a0654a61ae6 nodeName:}" failed. No retries permitted until 2026-07-24 16:43:48.031152257 +0000 UTC m=+50.673116118 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7960f9f4-815e-4b60-bd8c-2a0654a61ae6-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-mt8vp" (UID: "7960f9f4-815e-4b60-bd8c-2a0654a61ae6") : secret "networking-console-plugin-cert" not found
Jul 24 16:43:40.031418 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:40.031169    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/708abf59-5c2a-46e2-8506-386e91731078-metrics-tls podName:708abf59-5c2a-46e2-8506-386e91731078 nodeName:}" failed. No retries permitted until 2026-07-24 16:43:48.031163747 +0000 UTC m=+50.673127609 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/708abf59-5c2a-46e2-8506-386e91731078-metrics-tls") pod "dns-default-vrscm" (UID: "708abf59-5c2a-46e2-8506-386e91731078") : secret "dns-default-metrics-tls" not found
Jul 24 16:43:40.370459 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:40.370361    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:43:40.471117 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:40.471080    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:43:44.326171 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:44.326133    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:43:44.426346 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:44.426308    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:43:44.492214 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:44.492190    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:43:48.086664 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:48.086636    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/708abf59-5c2a-46e2-8506-386e91731078-metrics-tls\") pod \"dns-default-vrscm\" (UID: \"708abf59-5c2a-46e2-8506-386e91731078\") " pod="openshift-dns/dns-default-vrscm"
Jul 24 16:43:48.087115 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:48.086672    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-registry-tls\") pod \"image-registry-c8f78f7cf-2sdpb\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") " pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:43:48.087115 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:48.086737    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7960f9f4-815e-4b60-bd8c-2a0654a61ae6-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-mt8vp\" (UID: \"7960f9f4-815e-4b60-bd8c-2a0654a61ae6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-mt8vp"
Jul 24 16:43:48.087115 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:48.086789    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 16:43:48.087115 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:48.086844    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 16:43:48.087115 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:48.086861    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c8f78f7cf-2sdpb: secret "image-registry-tls" not found
Jul 24 16:43:48.087115 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:48.086877    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 16:43:48.087115 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:48.086852    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/708abf59-5c2a-46e2-8506-386e91731078-metrics-tls podName:708abf59-5c2a-46e2-8506-386e91731078 nodeName:}" failed. No retries permitted until 2026-07-24 16:44:04.086835542 +0000 UTC m=+66.728799405 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/708abf59-5c2a-46e2-8506-386e91731078-metrics-tls") pod "dns-default-vrscm" (UID: "708abf59-5c2a-46e2-8506-386e91731078") : secret "dns-default-metrics-tls" not found
Jul 24 16:43:48.087115 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:48.086908    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-registry-tls podName:6e7f3a6f-1da1-4a07-b736-512dfb2afd6a nodeName:}" failed. No retries permitted until 2026-07-24 16:44:04.086896082 +0000 UTC m=+66.728859943 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-registry-tls") pod "image-registry-c8f78f7cf-2sdpb" (UID: "6e7f3a6f-1da1-4a07-b736-512dfb2afd6a") : secret "image-registry-tls" not found
Jul 24 16:43:48.087115 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:43:48.086954    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7960f9f4-815e-4b60-bd8c-2a0654a61ae6-networking-console-plugin-cert podName:7960f9f4-815e-4b60-bd8c-2a0654a61ae6 nodeName:}" failed. No retries permitted until 2026-07-24 16:44:04.086918803 +0000 UTC m=+66.728882684 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7960f9f4-815e-4b60-bd8c-2a0654a61ae6-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-mt8vp" (UID: "7960f9f4-815e-4b60-bd8c-2a0654a61ae6") : secret "networking-console-plugin-cert" not found
Jul 24 16:43:48.235162 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:48.235123    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:43:48.291775 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:48.291745    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:43:51.962386 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:51.962343    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:43:52.063772 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:52.063735    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:43:52.465452 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:52.465410    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:43:52.563953 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:52.563903    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:43:58.240710 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:58.240681    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gt5dk"
Jul 24 16:43:58.401174 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:58.401133    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:43:58.499058 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:43:58.498985    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:44:00.817687 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:44:00.817655    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:44:00.916518 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:44:00.916485    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:44:02.590985 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:44:02.590914    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54f4b6d0-be12-40ae-b3ac-f7acb8e715b9-metrics-certs\") pod \"network-metrics-daemon-8lwz8\" (UID: \"54f4b6d0-be12-40ae-b3ac-f7acb8e715b9\") " pod="openshift-multus/network-metrics-daemon-8lwz8"
Jul 24 16:44:02.591390 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:44:02.591065    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/573b3eaf-dd83-456c-b7e0-35cb5aad6884-original-pull-secret\") pod \"global-pull-secret-syncer-vzbnm\" (UID: \"573b3eaf-dd83-456c-b7e0-35cb5aad6884\") " pod="kube-system/global-pull-secret-syncer-vzbnm"
Jul 24 16:44:02.593355 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:44:02.593336    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 16:44:02.593425 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:44:02.593336    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 16:44:02.601809 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:44:02.601790    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 24 16:44:02.601884 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:44:02.601845    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54f4b6d0-be12-40ae-b3ac-f7acb8e715b9-metrics-certs podName:54f4b6d0-be12-40ae-b3ac-f7acb8e715b9 nodeName:}" failed. No retries permitted until 2026-07-24 16:45:06.601828702 +0000 UTC m=+129.243792564 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54f4b6d0-be12-40ae-b3ac-f7acb8e715b9-metrics-certs") pod "network-metrics-daemon-8lwz8" (UID: "54f4b6d0-be12-40ae-b3ac-f7acb8e715b9") : secret "metrics-daemon-secret" not found
Jul 24 16:44:02.604991 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:44:02.604971    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/573b3eaf-dd83-456c-b7e0-35cb5aad6884-original-pull-secret\") pod \"global-pull-secret-syncer-vzbnm\" (UID: \"573b3eaf-dd83-456c-b7e0-35cb5aad6884\") " pod="kube-system/global-pull-secret-syncer-vzbnm"
Jul 24 16:44:02.692064 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:44:02.692028    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndwn5\" (UniqueName: \"kubernetes.io/projected/a12b44d3-5fd7-4543-bf4b-6b9ba880c11b-kube-api-access-ndwn5\") pod \"network-check-target-hhvqg\" (UID: \"a12b44d3-5fd7-4543-bf4b-6b9ba880c11b\") " pod="openshift-network-diagnostics/network-check-target-hhvqg"
Jul 24 16:44:02.694667 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:44:02.694647    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 16:44:02.705458 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:44:02.705442    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 16:44:02.715594 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:44:02.715577    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndwn5\" (UniqueName: \"kubernetes.io/projected/a12b44d3-5fd7-4543-bf4b-6b9ba880c11b-kube-api-access-ndwn5\") pod \"network-check-target-hhvqg\" (UID: \"a12b44d3-5fd7-4543-bf4b-6b9ba880c11b\") " pod="openshift-network-diagnostics/network-check-target-hhvqg"
Jul 24 16:44:02.808770 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:44:02.808741    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vzbnm"
Jul 24 16:44:02.819095 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:44:02.819071    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nn52b\""
Jul 24 16:44:02.827373 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:44:02.827354    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hhvqg"
Jul 24 16:44:02.942190 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:44:02.942161    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vzbnm"]
Jul 24 16:44:02.945276 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:44:02.945248    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod573b3eaf_dd83_456c_b7e0_35cb5aad6884.slice/crio-a2751fd23c2db03b76ec400f98a32c77f87961d51639298a957a443b1e920ed6 WatchSource:0}: Error finding container a2751fd23c2db03b76ec400f98a32c77f87961d51639298a957a443b1e920ed6: Status 404 returned error can't find the container with id a2751fd23c2db03b76ec400f98a32c77f87961d51639298a957a443b1e920ed6
Jul 24 16:44:02.956798 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:44:02.956775    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hhvqg"]
Jul 24 16:44:02.959873 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:44:02.959846    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda12b44d3_5fd7_4543_bf4b_6b9ba880c11b.slice/crio-4afd8948a5e7fe4d8dc92d5735db262eb7641cde58bf69b118c48f3becee6769 WatchSource:0}: Error finding container 4afd8948a5e7fe4d8dc92d5735db262eb7641cde58bf69b118c48f3becee6769: Status 404 returned error can't find the container with id 4afd8948a5e7fe4d8dc92d5735db262eb7641cde58bf69b118c48f3becee6769
Jul 24 16:44:03.320288 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:44:03.320252    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hhvqg" event={"ID":"a12b44d3-5fd7-4543-bf4b-6b9ba880c11b","Type":"ContainerStarted","Data":"4afd8948a5e7fe4d8dc92d5735db262eb7641cde58bf69b118c48f3becee6769"}
Jul 24 16:44:03.321242 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:44:03.321219    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vzbnm" event={"ID":"573b3eaf-dd83-456c-b7e0-35cb5aad6884","Type":"ContainerStarted","Data":"a2751fd23c2db03b76ec400f98a32c77f87961d51639298a957a443b1e920ed6"}
Jul 24 16:44:04.103864 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:44:04.103826    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/708abf59-5c2a-46e2-8506-386e91731078-metrics-tls\") pod \"dns-default-vrscm\" (UID: \"708abf59-5c2a-46e2-8506-386e91731078\") " pod="openshift-dns/dns-default-vrscm"
Jul 24 16:44:04.104317 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:44:04.103877    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-registry-tls\") pod \"image-registry-c8f78f7cf-2sdpb\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") " pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:44:04.104317 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:44:04.103960    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7960f9f4-815e-4b60-bd8c-2a0654a61ae6-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-mt8vp\" (UID: \"7960f9f4-815e-4b60-bd8c-2a0654a61ae6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-mt8vp"
Jul 24 16:44:04.104317 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:44:04.104002    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 16:44:04.104317 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:44:04.104039    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 16:44:04.104317 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:44:04.104055    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c8f78f7cf-2sdpb: secret "image-registry-tls" not found
Jul 24 16:44:04.104317 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:44:04.104059    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 16:44:04.104317 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:44:04.104073    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/708abf59-5c2a-46e2-8506-386e91731078-metrics-tls podName:708abf59-5c2a-46e2-8506-386e91731078 nodeName:}" failed. No retries permitted until 2026-07-24 16:44:36.10405711 +0000 UTC m=+98.746020976 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/708abf59-5c2a-46e2-8506-386e91731078-metrics-tls") pod "dns-default-vrscm" (UID: "708abf59-5c2a-46e2-8506-386e91731078") : secret "dns-default-metrics-tls" not found
Jul 24 16:44:04.104317 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:44:04.104102    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-registry-tls podName:6e7f3a6f-1da1-4a07-b736-512dfb2afd6a nodeName:}" failed. No retries permitted until 2026-07-24 16:44:36.104085785 +0000 UTC m=+98.746049662 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-registry-tls") pod "image-registry-c8f78f7cf-2sdpb" (UID: "6e7f3a6f-1da1-4a07-b736-512dfb2afd6a") : secret "image-registry-tls" not found
Jul 24 16:44:04.104317 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:44:04.104118    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7960f9f4-815e-4b60-bd8c-2a0654a61ae6-networking-console-plugin-cert podName:7960f9f4-815e-4b60-bd8c-2a0654a61ae6 nodeName:}" failed. No retries permitted until 2026-07-24 16:44:36.104111586 +0000 UTC m=+98.746075447 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7960f9f4-815e-4b60-bd8c-2a0654a61ae6-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-mt8vp" (UID: "7960f9f4-815e-4b60-bd8c-2a0654a61ae6") : secret "networking-console-plugin-cert" not found
Jul 24 16:44:08.336128 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:44:08.336036    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hhvqg" event={"ID":"a12b44d3-5fd7-4543-bf4b-6b9ba880c11b","Type":"ContainerStarted","Data":"118223288e79381e2547eb41126d9e42c5c42094d1b9ad2788176f694e9740b4"}
Jul 24 16:44:08.336128 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:44:08.336112    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hhvqg"
Jul 24 16:44:08.337411 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:44:08.337389    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vzbnm" event={"ID":"573b3eaf-dd83-456c-b7e0-35cb5aad6884","Type":"ContainerStarted","Data":"240cbb71ac499423421ae3a2125782c805fedccf41ead445e64fe96da9d1b15c"}
Jul 24 16:44:08.352001 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:44:08.351965    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hhvqg" podStartSLOduration=64.357016656 podStartE2EDuration="1m9.351953228s" totalImagesPullingTime="4.994936572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:42:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:44:08.351012957 +0000 UTC m=+70.992976841" watchObservedRunningTime="2026-07-24 16:44:08.351953228 +0000 UTC m=+70.993917122"
Jul 24 16:44:08.365143 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:44:08.365099    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vzbnm" podStartSLOduration=64.358024219 podStartE2EDuration="1m9.365085834s" totalImagesPullingTime="5.007061615s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 16:42:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:44:08.36483995 +0000 UTC m=+71.006803830" watchObservedRunningTime="2026-07-24 16:44:08.365085834 +0000 UTC m=+71.007049718"
Jul 24 16:44:13.927672 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:44:13.927635    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:44:13.982889 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:44:13.982858    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:44:14.036121 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:44:14.036095    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:44:36.130462 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:44:36.130317    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7960f9f4-815e-4b60-bd8c-2a0654a61ae6-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-mt8vp\" (UID: \"7960f9f4-815e-4b60-bd8c-2a0654a61ae6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-mt8vp"
Jul 24 16:44:36.130462 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:44:36.130390    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/708abf59-5c2a-46e2-8506-386e91731078-metrics-tls\") pod \"dns-default-vrscm\" (UID: \"708abf59-5c2a-46e2-8506-386e91731078\") " pod="openshift-dns/dns-default-vrscm"
Jul 24 16:44:36.130462 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:44:36.130407    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-registry-tls\") pod \"image-registry-c8f78f7cf-2sdpb\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") " pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:44:36.131067 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:44:36.130470    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 16:44:36.131067 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:44:36.130482    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 16:44:36.131067 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:44:36.130493    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c8f78f7cf-2sdpb: secret "image-registry-tls" not found
Jul 24 16:44:36.131067 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:44:36.130524    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 16:44:36.131067 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:44:36.130552    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7960f9f4-815e-4b60-bd8c-2a0654a61ae6-networking-console-plugin-cert podName:7960f9f4-815e-4b60-bd8c-2a0654a61ae6 nodeName:}" failed. No retries permitted until 2026-07-24 16:45:40.130532475 +0000 UTC m=+162.772496353 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7960f9f4-815e-4b60-bd8c-2a0654a61ae6-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-mt8vp" (UID: "7960f9f4-815e-4b60-bd8c-2a0654a61ae6") : secret "networking-console-plugin-cert" not found
Jul 24 16:44:36.131067 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:44:36.130572    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/708abf59-5c2a-46e2-8506-386e91731078-metrics-tls podName:708abf59-5c2a-46e2-8506-386e91731078 nodeName:}" failed. No retries permitted until 2026-07-24 16:45:40.130560613 +0000 UTC m=+162.772524474 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/708abf59-5c2a-46e2-8506-386e91731078-metrics-tls") pod "dns-default-vrscm" (UID: "708abf59-5c2a-46e2-8506-386e91731078") : secret "dns-default-metrics-tls" not found
Jul 24 16:44:36.131067 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:44:36.130586    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-registry-tls podName:6e7f3a6f-1da1-4a07-b736-512dfb2afd6a nodeName:}" failed. No retries permitted until 2026-07-24 16:45:40.130580244 +0000 UTC m=+162.772544106 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-registry-tls") pod "image-registry-c8f78f7cf-2sdpb" (UID: "6e7f3a6f-1da1-4a07-b736-512dfb2afd6a") : secret "image-registry-tls" not found
Jul 24 16:44:38.104081 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:44:38.104036    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:44:39.341918 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:44:39.341891    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hhvqg"
Jul 24 16:45:06.653953 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:45:06.653896    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54f4b6d0-be12-40ae-b3ac-f7acb8e715b9-metrics-certs\") pod \"network-metrics-daemon-8lwz8\" (UID: \"54f4b6d0-be12-40ae-b3ac-f7acb8e715b9\") " pod="openshift-multus/network-metrics-daemon-8lwz8"
Jul 24 16:45:06.654455 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:45:06.654051    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 24 16:45:06.654455 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:45:06.654126    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54f4b6d0-be12-40ae-b3ac-f7acb8e715b9-metrics-certs podName:54f4b6d0-be12-40ae-b3ac-f7acb8e715b9 nodeName:}" failed. No retries permitted until 2026-07-24 16:47:08.654109153 +0000 UTC m=+251.296073014 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54f4b6d0-be12-40ae-b3ac-f7acb8e715b9-metrics-certs") pod "network-metrics-daemon-8lwz8" (UID: "54f4b6d0-be12-40ae-b3ac-f7acb8e715b9") : secret "metrics-daemon-secret" not found
Jul 24 16:45:35.244115 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:45:35.244049    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-74c9f884b7-mt8vp" podUID="7960f9f4-815e-4b60-bd8c-2a0654a61ae6"
Jul 24 16:45:35.253209 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:45:35.253177    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb" podUID="6e7f3a6f-1da1-4a07-b736-512dfb2afd6a"
Jul 24 16:45:35.261306 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:45:35.261287    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-vrscm" podUID="708abf59-5c2a-46e2-8506-386e91731078"
Jul 24 16:45:35.541682 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:45:35.541607    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vrscm"
Jul 24 16:45:35.541810 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:45:35.541607    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-mt8vp"
Jul 24 16:45:35.541810 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:45:35.541615    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:45:37.022759 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:45:37.022720    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-8lwz8" podUID="54f4b6d0-be12-40ae-b3ac-f7acb8e715b9"
Jul 24 16:45:40.202078 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:45:40.202040    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/708abf59-5c2a-46e2-8506-386e91731078-metrics-tls\") pod \"dns-default-vrscm\" (UID: \"708abf59-5c2a-46e2-8506-386e91731078\") " pod="openshift-dns/dns-default-vrscm"
Jul 24 16:45:40.202078 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:45:40.202080    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-registry-tls\") pod \"image-registry-c8f78f7cf-2sdpb\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") " pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:45:40.202554 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:45:40.202135    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7960f9f4-815e-4b60-bd8c-2a0654a61ae6-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-mt8vp\" (UID: \"7960f9f4-815e-4b60-bd8c-2a0654a61ae6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-mt8vp"
Jul 24 16:45:40.202554 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:45:40.202215    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 16:45:40.202554 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:45:40.202287    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7960f9f4-815e-4b60-bd8c-2a0654a61ae6-networking-console-plugin-cert podName:7960f9f4-815e-4b60-bd8c-2a0654a61ae6 nodeName:}" failed. No retries permitted until 2026-07-24 16:47:42.202272033 +0000 UTC m=+284.844235895 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7960f9f4-815e-4b60-bd8c-2a0654a61ae6-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-mt8vp" (UID: "7960f9f4-815e-4b60-bd8c-2a0654a61ae6") : secret "networking-console-plugin-cert" not found
Jul 24 16:45:40.202554 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:45:40.202215    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 16:45:40.202554 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:45:40.202323    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 16:45:40.202554 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:45:40.202350    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/708abf59-5c2a-46e2-8506-386e91731078-metrics-tls podName:708abf59-5c2a-46e2-8506-386e91731078 nodeName:}" failed. No retries permitted until 2026-07-24 16:47:42.2023377 +0000 UTC m=+284.844301568 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/708abf59-5c2a-46e2-8506-386e91731078-metrics-tls") pod "dns-default-vrscm" (UID: "708abf59-5c2a-46e2-8506-386e91731078") : secret "dns-default-metrics-tls" not found
Jul 24 16:45:40.202554 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:45:40.202349    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c8f78f7cf-2sdpb: secret "image-registry-tls" not found
Jul 24 16:45:40.202554 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:45:40.202387    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-registry-tls podName:6e7f3a6f-1da1-4a07-b736-512dfb2afd6a nodeName:}" failed. No retries permitted until 2026-07-24 16:47:42.202378643 +0000 UTC m=+284.844342506 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-registry-tls") pod "image-registry-c8f78f7cf-2sdpb" (UID: "6e7f3a6f-1da1-4a07-b736-512dfb2afd6a") : secret "image-registry-tls" not found
Jul 24 16:45:48.000334 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:45:48.000306    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lwz8"
Jul 24 16:46:53.102514 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:46:53.102484    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:08.711604 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:08.711552    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54f4b6d0-be12-40ae-b3ac-f7acb8e715b9-metrics-certs\") pod \"network-metrics-daemon-8lwz8\" (UID: \"54f4b6d0-be12-40ae-b3ac-f7acb8e715b9\") " pod="openshift-multus/network-metrics-daemon-8lwz8"
Jul 24 16:47:08.712108 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:08.711678    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 24 16:47:08.712108 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:08.711732    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54f4b6d0-be12-40ae-b3ac-f7acb8e715b9-metrics-certs podName:54f4b6d0-be12-40ae-b3ac-f7acb8e715b9 nodeName:}" failed. No retries permitted until 2026-07-24 16:49:10.711718652 +0000 UTC m=+373.353682513 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54f4b6d0-be12-40ae-b3ac-f7acb8e715b9-metrics-certs") pod "network-metrics-daemon-8lwz8" (UID: "54f4b6d0-be12-40ae-b3ac-f7acb8e715b9") : secret "metrics-daemon-secret" not found
Jul 24 16:47:25.063951 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.063854    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-4rmnz"]
Jul 24 16:47:25.066731 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.066670    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-4rmnz"
Jul 24 16:47:25.067347 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.067200    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-62xxb"]
Jul 24 16:47:25.069147 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.069111    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-tb4s6\""
Jul 24 16:47:25.069913 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.069899    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-ddklx"]
Jul 24 16:47:25.070029 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.070016    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-62xxb"
Jul 24 16:47:25.072171 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.072122    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-phfhr\""
Jul 24 16:47:25.072337 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.072318    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 24 16:47:25.072384 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.072350    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 24 16:47:25.072582 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.072567    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-ddklx"
Jul 24 16:47:25.075191 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.075175    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 24 16:47:25.076279 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.076258    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 24 16:47:25.076362 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.076280    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 24 16:47:25.077382 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.077366    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-jcglq\""
Jul 24 16:47:25.077457 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.077427    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 24 16:47:25.081921 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.081898    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-4rmnz"]
Jul 24 16:47:25.085908 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.084413    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-ddklx"]
Jul 24 16:47:25.086717 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.086692    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 24 16:47:25.097136 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.097110    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-62xxb"]
Jul 24 16:47:25.121866 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.121832    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fb5a9838-1e05-4f86-b46c-760bd7aaa038-trusted-ca\") pod \"console-operator-584dcb6b54-ddklx\" (UID: \"fb5a9838-1e05-4f86-b46c-760bd7aaa038\") " pod="openshift-console-operator/console-operator-584dcb6b54-ddklx"
Jul 24 16:47:25.121866 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.121868    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw5gs\" (UniqueName: \"kubernetes.io/projected/1f6e96ed-11c0-4dbb-afb0-231d599a870b-kube-api-access-dw5gs\") pod \"volume-data-source-validator-549c6d54cf-62xxb\" (UID: \"1f6e96ed-11c0-4dbb-afb0-231d599a870b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-62xxb"
Jul 24 16:47:25.122127 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.121890    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb5a9838-1e05-4f86-b46c-760bd7aaa038-serving-cert\") pod \"console-operator-584dcb6b54-ddklx\" (UID: \"fb5a9838-1e05-4f86-b46c-760bd7aaa038\") " pod="openshift-console-operator/console-operator-584dcb6b54-ddklx"
Jul 24 16:47:25.122127 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.121913    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fb5a9838-1e05-4f86-b46c-760bd7aaa038-config\") pod \"console-operator-584dcb6b54-ddklx\" (UID: \"fb5a9838-1e05-4f86-b46c-760bd7aaa038\") " pod="openshift-console-operator/console-operator-584dcb6b54-ddklx"
Jul 24 16:47:25.122127 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.121953    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbrkf\" (UniqueName: \"kubernetes.io/projected/20548047-6c72-4740-bef0-b49e84fa8f37-kube-api-access-rbrkf\") pod \"network-check-source-7d88cb55d8-4rmnz\" (UID: \"20548047-6c72-4740-bef0-b49e84fa8f37\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-4rmnz"
Jul 24 16:47:25.122127 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.122067    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x98h\" (UniqueName: \"kubernetes.io/projected/fb5a9838-1e05-4f86-b46c-760bd7aaa038-kube-api-access-7x98h\") pod \"console-operator-584dcb6b54-ddklx\" (UID: \"fb5a9838-1e05-4f86-b46c-760bd7aaa038\") " pod="openshift-console-operator/console-operator-584dcb6b54-ddklx"
Jul 24 16:47:25.164260 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.164221    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xl9vk"]
Jul 24 16:47:25.167231 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.167214    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xl9vk"
Jul 24 16:47:25.169647 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.169626    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 24 16:47:25.169740 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.169647    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-9r4rd\""
Jul 24 16:47:25.170031 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.170018    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 24 16:47:25.170141 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.170126    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 24 16:47:25.170228 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.170209    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 24 16:47:25.171896 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.171875    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-5qj6l"]
Jul 24 16:47:25.174559 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.174541    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-746d84cf85-s4sxm"]
Jul 24 16:47:25.174689 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.174672    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-5qj6l"
Jul 24 16:47:25.176865 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.176848    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 24 16:47:25.177344 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.177324    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 24 16:47:25.177481 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.177372    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 24 16:47:25.177481 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.177440    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-746d84cf85-s4sxm"
Jul 24 16:47:25.177481 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.177371    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wvdgz"]
Jul 24 16:47:25.177481 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.177444    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-j9sh9\""
Jul 24 16:47:25.177914 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.177843    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 24 16:47:25.180453 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.180430    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xl9vk"]
Jul 24 16:47:25.180565 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.180434    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 24 16:47:25.180565 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.180469    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 24 16:47:25.180673 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.180594    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wvdgz"
Jul 24 16:47:25.180771 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.180753    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lkfm4\""
Jul 24 16:47:25.180863 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.180764    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 24 16:47:25.180987 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.180869    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 24 16:47:25.180987 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.180843    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 24 16:47:25.181256 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.181235    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 24 16:47:25.182827 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.182807    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 24 16:47:25.183527 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.183508    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 24 16:47:25.183627 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.183527    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-zjkf9\""
Jul 24 16:47:25.183627 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.183619    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 24 16:47:25.183819 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.183804    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 24 16:47:25.187722 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.187700    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wvdgz"]
Jul 24 16:47:25.193297 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.193273    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-746d84cf85-s4sxm"]
Jul 24 16:47:25.194295 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.194273    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-5qj6l"]
Jul 24 16:47:25.222611 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.222576    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fb5a9838-1e05-4f86-b46c-760bd7aaa038-config\") pod \"console-operator-584dcb6b54-ddklx\" (UID: \"fb5a9838-1e05-4f86-b46c-760bd7aaa038\") " pod="openshift-console-operator/console-operator-584dcb6b54-ddklx"
Jul 24 16:47:25.222611 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.222616    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbrkf\" (UniqueName: \"kubernetes.io/projected/20548047-6c72-4740-bef0-b49e84fa8f37-kube-api-access-rbrkf\") pod \"network-check-source-7d88cb55d8-4rmnz\" (UID: \"20548047-6c72-4740-bef0-b49e84fa8f37\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-4rmnz"
Jul 24 16:47:25.222824 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.222637    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92x72\" (UniqueName: \"kubernetes.io/projected/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-kube-api-access-92x72\") pod \"router-default-746d84cf85-s4sxm\" (UID: \"f3934ff0-8fae-4d73-82e3-96276d9ff5b0\") " pod="openshift-ingress/router-default-746d84cf85-s4sxm"
Jul 24 16:47:25.222824 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.222667    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3460f62-6389-4673-9e33-e4e5aa0d4d9a-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-xl9vk\" (UID: \"e3460f62-6389-4673-9e33-e4e5aa0d4d9a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xl9vk"
Jul 24 16:47:25.222824 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.222719    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxvrf\" (UniqueName: \"kubernetes.io/projected/ef1872b5-49d6-4162-bcba-afea4c9ca621-kube-api-access-bxvrf\") pod \"insights-operator-5885fdb759-5qj6l\" (UID: \"ef1872b5-49d6-4162-bcba-afea4c9ca621\") " pod="openshift-insights/insights-operator-5885fdb759-5qj6l"
Jul 24 16:47:25.222824 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.222768    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fb5a9838-1e05-4f86-b46c-760bd7aaa038-trusted-ca\") pod \"console-operator-584dcb6b54-ddklx\" (UID: \"fb5a9838-1e05-4f86-b46c-760bd7aaa038\") " pod="openshift-console-operator/console-operator-584dcb6b54-ddklx"
Jul 24 16:47:25.222824 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.222797    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef1872b5-49d6-4162-bcba-afea4c9ca621-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-5qj6l\" (UID: \"ef1872b5-49d6-4162-bcba-afea4c9ca621\") " pod="openshift-insights/insights-operator-5885fdb759-5qj6l"
Jul 24 16:47:25.222824 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.222823    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw5gs\" (UniqueName: \"kubernetes.io/projected/1f6e96ed-11c0-4dbb-afb0-231d599a870b-kube-api-access-dw5gs\") pod \"volume-data-source-validator-549c6d54cf-62xxb\" (UID: \"1f6e96ed-11c0-4dbb-afb0-231d599a870b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-62xxb"
Jul 24 16:47:25.223098 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.222873    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht5j6\" (UniqueName: \"kubernetes.io/projected/d185fb45-2628-4ccc-8afb-fa4ab0b344c6-kube-api-access-ht5j6\") pod \"cluster-samples-operator-6748489656-wvdgz\" (UID: \"d185fb45-2628-4ccc-8afb-fa4ab0b344c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wvdgz"
Jul 24 16:47:25.223098 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.222921    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ef1872b5-49d6-4162-bcba-afea4c9ca621-snapshots\") pod \"insights-operator-5885fdb759-5qj6l\" (UID: \"ef1872b5-49d6-4162-bcba-afea4c9ca621\") " pod="openshift-insights/insights-operator-5885fdb759-5qj6l"
Jul 24 16:47:25.223098 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.223034    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-default-certificate\") pod \"router-default-746d84cf85-s4sxm\" (UID: \"f3934ff0-8fae-4d73-82e3-96276d9ff5b0\") " pod="openshift-ingress/router-default-746d84cf85-s4sxm"
Jul 24 16:47:25.223098 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.223083    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x98h\" (UniqueName: \"kubernetes.io/projected/fb5a9838-1e05-4f86-b46c-760bd7aaa038-kube-api-access-7x98h\") pod \"console-operator-584dcb6b54-ddklx\" (UID: \"fb5a9838-1e05-4f86-b46c-760bd7aaa038\") " pod="openshift-console-operator/console-operator-584dcb6b54-ddklx"
Jul 24 16:47:25.223282 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.223105    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d185fb45-2628-4ccc-8afb-fa4ab0b344c6-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-wvdgz\" (UID: \"d185fb45-2628-4ccc-8afb-fa4ab0b344c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wvdgz"
Jul 24 16:47:25.223282 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.223127    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-service-ca-bundle\") pod \"router-default-746d84cf85-s4sxm\" (UID: \"f3934ff0-8fae-4d73-82e3-96276d9ff5b0\") " pod="openshift-ingress/router-default-746d84cf85-s4sxm"
Jul 24 16:47:25.223282 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.223145    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ef1872b5-49d6-4162-bcba-afea4c9ca621-serving-cert\") pod \"insights-operator-5885fdb759-5qj6l\" (UID: \"ef1872b5-49d6-4162-bcba-afea4c9ca621\") " pod="openshift-insights/insights-operator-5885fdb759-5qj6l"
Jul 24 16:47:25.223282 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.223191    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-metrics-certs\") pod \"router-default-746d84cf85-s4sxm\" (UID: \"f3934ff0-8fae-4d73-82e3-96276d9ff5b0\") " pod="openshift-ingress/router-default-746d84cf85-s4sxm"
Jul 24 16:47:25.223282 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.223234    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3460f62-6389-4673-9e33-e4e5aa0d4d9a-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-xl9vk\" (UID: \"e3460f62-6389-4673-9e33-e4e5aa0d4d9a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xl9vk"
Jul 24 16:47:25.223282 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.223264    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef1872b5-49d6-4162-bcba-afea4c9ca621-tmp\") pod \"insights-operator-5885fdb759-5qj6l\" (UID: \"ef1872b5-49d6-4162-bcba-afea4c9ca621\") " pod="openshift-insights/insights-operator-5885fdb759-5qj6l"
Jul 24 16:47:25.223504 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.223293    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wzx2\" (UniqueName: \"kubernetes.io/projected/e3460f62-6389-4673-9e33-e4e5aa0d4d9a-kube-api-access-4wzx2\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-xl9vk\" (UID: \"e3460f62-6389-4673-9e33-e4e5aa0d4d9a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xl9vk"
Jul 24 16:47:25.223504 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.223322    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fb5a9838-1e05-4f86-b46c-760bd7aaa038-config\") pod \"console-operator-584dcb6b54-ddklx\" (UID: \"fb5a9838-1e05-4f86-b46c-760bd7aaa038\") " pod="openshift-console-operator/console-operator-584dcb6b54-ddklx"
Jul 24 16:47:25.223504 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.223324    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef1872b5-49d6-4162-bcba-afea4c9ca621-service-ca-bundle\") pod \"insights-operator-5885fdb759-5qj6l\" (UID: \"ef1872b5-49d6-4162-bcba-afea4c9ca621\") " pod="openshift-insights/insights-operator-5885fdb759-5qj6l"
Jul 24 16:47:25.223504 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.223378    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb5a9838-1e05-4f86-b46c-760bd7aaa038-serving-cert\") pod \"console-operator-584dcb6b54-ddklx\" (UID: \"fb5a9838-1e05-4f86-b46c-760bd7aaa038\") " pod="openshift-console-operator/console-operator-584dcb6b54-ddklx"
Jul 24 16:47:25.223504 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.223397    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-stats-auth\") pod \"router-default-746d84cf85-s4sxm\" (UID: \"f3934ff0-8fae-4d73-82e3-96276d9ff5b0\") " pod="openshift-ingress/router-default-746d84cf85-s4sxm"
Jul 24 16:47:25.223652 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.223586    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fb5a9838-1e05-4f86-b46c-760bd7aaa038-trusted-ca\") pod \"console-operator-584dcb6b54-ddklx\" (UID: \"fb5a9838-1e05-4f86-b46c-760bd7aaa038\") " pod="openshift-console-operator/console-operator-584dcb6b54-ddklx"
Jul 24 16:47:25.225980 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.225956    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb5a9838-1e05-4f86-b46c-760bd7aaa038-serving-cert\") pod \"console-operator-584dcb6b54-ddklx\" (UID: \"fb5a9838-1e05-4f86-b46c-760bd7aaa038\") " pod="openshift-console-operator/console-operator-584dcb6b54-ddklx"
Jul 24 16:47:25.231127 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.231102    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x98h\" (UniqueName: \"kubernetes.io/projected/fb5a9838-1e05-4f86-b46c-760bd7aaa038-kube-api-access-7x98h\") pod \"console-operator-584dcb6b54-ddklx\" (UID: \"fb5a9838-1e05-4f86-b46c-760bd7aaa038\") " pod="openshift-console-operator/console-operator-584dcb6b54-ddklx"
Jul 24 16:47:25.231249 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.231171    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw5gs\" (UniqueName: \"kubernetes.io/projected/1f6e96ed-11c0-4dbb-afb0-231d599a870b-kube-api-access-dw5gs\") pod \"volume-data-source-validator-549c6d54cf-62xxb\" (UID: \"1f6e96ed-11c0-4dbb-afb0-231d599a870b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-62xxb"
Jul 24 16:47:25.231249 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.231245    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbrkf\" (UniqueName: \"kubernetes.io/projected/20548047-6c72-4740-bef0-b49e84fa8f37-kube-api-access-rbrkf\") pod \"network-check-source-7d88cb55d8-4rmnz\" (UID: \"20548047-6c72-4740-bef0-b49e84fa8f37\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-4rmnz"
Jul 24 16:47:25.308591 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.308555    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:25.324712 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.324652    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef1872b5-49d6-4162-bcba-afea4c9ca621-service-ca-bundle\") pod \"insights-operator-5885fdb759-5qj6l\" (UID: \"ef1872b5-49d6-4162-bcba-afea4c9ca621\") " pod="openshift-insights/insights-operator-5885fdb759-5qj6l"
Jul 24 16:47:25.324712 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.324685    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-stats-auth\") pod \"router-default-746d84cf85-s4sxm\" (UID: \"f3934ff0-8fae-4d73-82e3-96276d9ff5b0\") " pod="openshift-ingress/router-default-746d84cf85-s4sxm"
Jul 24 16:47:25.324712 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.324707    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92x72\" (UniqueName: \"kubernetes.io/projected/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-kube-api-access-92x72\") pod \"router-default-746d84cf85-s4sxm\" (UID: \"f3934ff0-8fae-4d73-82e3-96276d9ff5b0\") " pod="openshift-ingress/router-default-746d84cf85-s4sxm"
Jul 24 16:47:25.324910 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.324831    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3460f62-6389-4673-9e33-e4e5aa0d4d9a-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-xl9vk\" (UID: \"e3460f62-6389-4673-9e33-e4e5aa0d4d9a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xl9vk"
Jul 24 16:47:25.324910 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.324871    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxvrf\" (UniqueName: \"kubernetes.io/projected/ef1872b5-49d6-4162-bcba-afea4c9ca621-kube-api-access-bxvrf\") pod \"insights-operator-5885fdb759-5qj6l\" (UID: \"ef1872b5-49d6-4162-bcba-afea4c9ca621\") " pod="openshift-insights/insights-operator-5885fdb759-5qj6l"
Jul 24 16:47:25.325061 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.324952    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef1872b5-49d6-4162-bcba-afea4c9ca621-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-5qj6l\" (UID: \"ef1872b5-49d6-4162-bcba-afea4c9ca621\") " pod="openshift-insights/insights-operator-5885fdb759-5qj6l"
Jul 24 16:47:25.325061 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.324987    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht5j6\" (UniqueName: \"kubernetes.io/projected/d185fb45-2628-4ccc-8afb-fa4ab0b344c6-kube-api-access-ht5j6\") pod \"cluster-samples-operator-6748489656-wvdgz\" (UID: \"d185fb45-2628-4ccc-8afb-fa4ab0b344c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wvdgz"
Jul 24 16:47:25.325061 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.325014    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ef1872b5-49d6-4162-bcba-afea4c9ca621-snapshots\") pod \"insights-operator-5885fdb759-5qj6l\" (UID: \"ef1872b5-49d6-4162-bcba-afea4c9ca621\") " pod="openshift-insights/insights-operator-5885fdb759-5qj6l"
Jul 24 16:47:25.325061 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.325047    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-default-certificate\") pod \"router-default-746d84cf85-s4sxm\" (UID: \"f3934ff0-8fae-4d73-82e3-96276d9ff5b0\") " pod="openshift-ingress/router-default-746d84cf85-s4sxm"
Jul 24 16:47:25.325263 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.325102    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d185fb45-2628-4ccc-8afb-fa4ab0b344c6-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-wvdgz\" (UID: \"d185fb45-2628-4ccc-8afb-fa4ab0b344c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wvdgz"
Jul 24 16:47:25.325263 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.325125    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-service-ca-bundle\") pod \"router-default-746d84cf85-s4sxm\" (UID: \"f3934ff0-8fae-4d73-82e3-96276d9ff5b0\") " pod="openshift-ingress/router-default-746d84cf85-s4sxm"
Jul 24 16:47:25.325263 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.325151    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ef1872b5-49d6-4162-bcba-afea4c9ca621-serving-cert\") pod \"insights-operator-5885fdb759-5qj6l\" (UID: \"ef1872b5-49d6-4162-bcba-afea4c9ca621\") " pod="openshift-insights/insights-operator-5885fdb759-5qj6l"
Jul 24 16:47:25.325263 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.325181    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-metrics-certs\") pod \"router-default-746d84cf85-s4sxm\" (UID: \"f3934ff0-8fae-4d73-82e3-96276d9ff5b0\") " pod="openshift-ingress/router-default-746d84cf85-s4sxm"
Jul 24 16:47:25.325263 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.325209    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3460f62-6389-4673-9e33-e4e5aa0d4d9a-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-xl9vk\" (UID: \"e3460f62-6389-4673-9e33-e4e5aa0d4d9a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xl9vk"
Jul 24 16:47:25.325263 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.325235    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef1872b5-49d6-4162-bcba-afea4c9ca621-tmp\") pod \"insights-operator-5885fdb759-5qj6l\" (UID: \"ef1872b5-49d6-4162-bcba-afea4c9ca621\") " pod="openshift-insights/insights-operator-5885fdb759-5qj6l"
Jul 24 16:47:25.325542 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.325266    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wzx2\" (UniqueName: \"kubernetes.io/projected/e3460f62-6389-4673-9e33-e4e5aa0d4d9a-kube-api-access-4wzx2\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-xl9vk\" (UID: \"e3460f62-6389-4673-9e33-e4e5aa0d4d9a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xl9vk"
Jul 24 16:47:25.325542 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.325372    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef1872b5-49d6-4162-bcba-afea4c9ca621-service-ca-bundle\") pod \"insights-operator-5885fdb759-5qj6l\" (UID: \"ef1872b5-49d6-4162-bcba-afea4c9ca621\") " pod="openshift-insights/insights-operator-5885fdb759-5qj6l"
Jul 24 16:47:25.325542 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:25.325473    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 24 16:47:25.325718 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.325691    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef1872b5-49d6-4162-bcba-afea4c9ca621-tmp\") pod \"insights-operator-5885fdb759-5qj6l\" (UID: \"ef1872b5-49d6-4162-bcba-afea4c9ca621\") " pod="openshift-insights/insights-operator-5885fdb759-5qj6l"
Jul 24 16:47:25.325793 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:25.325776    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-metrics-certs podName:f3934ff0-8fae-4d73-82e3-96276d9ff5b0 nodeName:}" failed. No retries permitted until 2026-07-24 16:47:25.825708111 +0000 UTC m=+268.467671979 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-metrics-certs") pod "router-default-746d84cf85-s4sxm" (UID: "f3934ff0-8fae-4d73-82e3-96276d9ff5b0") : secret "router-metrics-certs-default" not found
Jul 24 16:47:25.325856 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:25.325810    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 24 16:47:25.325906 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:25.325868    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d185fb45-2628-4ccc-8afb-fa4ab0b344c6-samples-operator-tls podName:d185fb45-2628-4ccc-8afb-fa4ab0b344c6 nodeName:}" failed. No retries permitted until 2026-07-24 16:47:25.825851257 +0000 UTC m=+268.467815122 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d185fb45-2628-4ccc-8afb-fa4ab0b344c6-samples-operator-tls") pod "cluster-samples-operator-6748489656-wvdgz" (UID: "d185fb45-2628-4ccc-8afb-fa4ab0b344c6") : secret "samples-operator-tls" not found
Jul 24 16:47:25.325987 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:25.325957    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-service-ca-bundle podName:f3934ff0-8fae-4d73-82e3-96276d9ff5b0 nodeName:}" failed. No retries permitted until 2026-07-24 16:47:25.825929455 +0000 UTC m=+268.467893330 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-service-ca-bundle") pod "router-default-746d84cf85-s4sxm" (UID: "f3934ff0-8fae-4d73-82e3-96276d9ff5b0") : configmap references non-existent config key: service-ca.crt
Jul 24 16:47:25.326097 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.326077    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ef1872b5-49d6-4162-bcba-afea4c9ca621-snapshots\") pod \"insights-operator-5885fdb759-5qj6l\" (UID: \"ef1872b5-49d6-4162-bcba-afea4c9ca621\") " pod="openshift-insights/insights-operator-5885fdb759-5qj6l"
Jul 24 16:47:25.326138 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.326099    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3460f62-6389-4673-9e33-e4e5aa0d4d9a-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-xl9vk\" (UID: \"e3460f62-6389-4673-9e33-e4e5aa0d4d9a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xl9vk"
Jul 24 16:47:25.326640 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.326618    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef1872b5-49d6-4162-bcba-afea4c9ca621-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-5qj6l\" (UID: \"ef1872b5-49d6-4162-bcba-afea4c9ca621\") " pod="openshift-insights/insights-operator-5885fdb759-5qj6l"
Jul 24 16:47:25.327563 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.327539    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ef1872b5-49d6-4162-bcba-afea4c9ca621-serving-cert\") pod \"insights-operator-5885fdb759-5qj6l\" (UID: \"ef1872b5-49d6-4162-bcba-afea4c9ca621\") " pod="openshift-insights/insights-operator-5885fdb759-5qj6l"
Jul 24 16:47:25.327738 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.327706    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3460f62-6389-4673-9e33-e4e5aa0d4d9a-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-xl9vk\" (UID: \"e3460f62-6389-4673-9e33-e4e5aa0d4d9a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xl9vk"
Jul 24 16:47:25.327877 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.327862    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-default-certificate\") pod \"router-default-746d84cf85-s4sxm\" (UID: \"f3934ff0-8fae-4d73-82e3-96276d9ff5b0\") " pod="openshift-ingress/router-default-746d84cf85-s4sxm"
Jul 24 16:47:25.327969 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.327930    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-stats-auth\") pod \"router-default-746d84cf85-s4sxm\" (UID: \"f3934ff0-8fae-4d73-82e3-96276d9ff5b0\") " pod="openshift-ingress/router-default-746d84cf85-s4sxm"
Jul 24 16:47:25.332972 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.332922    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht5j6\" (UniqueName: \"kubernetes.io/projected/d185fb45-2628-4ccc-8afb-fa4ab0b344c6-kube-api-access-ht5j6\") pod \"cluster-samples-operator-6748489656-wvdgz\" (UID: \"d185fb45-2628-4ccc-8afb-fa4ab0b344c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wvdgz"
Jul 24 16:47:25.333457 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.333434    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxvrf\" (UniqueName: \"kubernetes.io/projected/ef1872b5-49d6-4162-bcba-afea4c9ca621-kube-api-access-bxvrf\") pod \"insights-operator-5885fdb759-5qj6l\" (UID: \"ef1872b5-49d6-4162-bcba-afea4c9ca621\") " pod="openshift-insights/insights-operator-5885fdb759-5qj6l"
Jul 24 16:47:25.334840 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.334820    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92x72\" (UniqueName: \"kubernetes.io/projected/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-kube-api-access-92x72\") pod \"router-default-746d84cf85-s4sxm\" (UID: \"f3934ff0-8fae-4d73-82e3-96276d9ff5b0\") " pod="openshift-ingress/router-default-746d84cf85-s4sxm"
Jul 24 16:47:25.334930 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.334852    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wzx2\" (UniqueName: \"kubernetes.io/projected/e3460f62-6389-4673-9e33-e4e5aa0d4d9a-kube-api-access-4wzx2\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-xl9vk\" (UID: \"e3460f62-6389-4673-9e33-e4e5aa0d4d9a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xl9vk"
Jul 24 16:47:25.376772 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.376727    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-4rmnz"
Jul 24 16:47:25.386729 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.386697    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-62xxb"
Jul 24 16:47:25.392783 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.392761    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-ddklx"
Jul 24 16:47:25.411244 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.411208    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:25.477590 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.477391    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xl9vk"
Jul 24 16:47:25.486887 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.486864    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-5qj6l"
Jul 24 16:47:25.531329 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.531243    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-4rmnz"]
Jul 24 16:47:25.538887 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:47:25.538853    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20548047_6c72_4740_bef0_b49e84fa8f37.slice/crio-404871f565b654cb1f6daadfb985e7ff557b32ce730934385c4b1f15d29d4333 WatchSource:0}: Error finding container 404871f565b654cb1f6daadfb985e7ff557b32ce730934385c4b1f15d29d4333: Status 404 returned error can't find the container with id 404871f565b654cb1f6daadfb985e7ff557b32ce730934385c4b1f15d29d4333
Jul 24 16:47:25.629685 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.629484    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xl9vk"]
Jul 24 16:47:25.641193 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.641166    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-5qj6l"]
Jul 24 16:47:25.645193 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:47:25.645171    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef1872b5_49d6_4162_bcba_afea4c9ca621.slice/crio-65c6d95af1b39732ac8b2bd611682bbd0ea772547dd33d771509d917f8786754 WatchSource:0}: Error finding container 65c6d95af1b39732ac8b2bd611682bbd0ea772547dd33d771509d917f8786754: Status 404 returned error can't find the container with id 65c6d95af1b39732ac8b2bd611682bbd0ea772547dd33d771509d917f8786754
Jul 24 16:47:25.752202 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.752170    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-62xxb"]
Jul 24 16:47:25.756082 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:47:25.756056    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f6e96ed_11c0_4dbb_afb0_231d599a870b.slice/crio-aff71bbcdf760d2dc1792199cf9fe7949bf8a9627d15e2647e360568c2ec53dd WatchSource:0}: Error finding container aff71bbcdf760d2dc1792199cf9fe7949bf8a9627d15e2647e360568c2ec53dd: Status 404 returned error can't find the container with id aff71bbcdf760d2dc1792199cf9fe7949bf8a9627d15e2647e360568c2ec53dd
Jul 24 16:47:25.756201 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.756151    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-ddklx"]
Jul 24 16:47:25.760475 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:47:25.760450    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb5a9838_1e05_4f86_b46c_760bd7aaa038.slice/crio-8cd9b2a92fb6b880dfa38c8f4f01c380a5e1ff75f190fa6fcd2ad380fb1df958 WatchSource:0}: Error finding container 8cd9b2a92fb6b880dfa38c8f4f01c380a5e1ff75f190fa6fcd2ad380fb1df958: Status 404 returned error can't find the container with id 8cd9b2a92fb6b880dfa38c8f4f01c380a5e1ff75f190fa6fcd2ad380fb1df958
Jul 24 16:47:25.795219 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.795173    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-62xxb" event={"ID":"1f6e96ed-11c0-4dbb-afb0-231d599a870b","Type":"ContainerStarted","Data":"aff71bbcdf760d2dc1792199cf9fe7949bf8a9627d15e2647e360568c2ec53dd"}
Jul 24 16:47:25.796261 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.796236    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xl9vk" event={"ID":"e3460f62-6389-4673-9e33-e4e5aa0d4d9a","Type":"ContainerStarted","Data":"aa1bcd1808d6b36125fd45acb89f28b2c57f2c001db390ea6cb5d1c8230ee024"}
Jul 24 16:47:25.797286 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.797265    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-ddklx" event={"ID":"fb5a9838-1e05-4f86-b46c-760bd7aaa038","Type":"ContainerStarted","Data":"8cd9b2a92fb6b880dfa38c8f4f01c380a5e1ff75f190fa6fcd2ad380fb1df958"}
Jul 24 16:47:25.798386 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.798364    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-5qj6l" event={"ID":"ef1872b5-49d6-4162-bcba-afea4c9ca621","Type":"ContainerStarted","Data":"65c6d95af1b39732ac8b2bd611682bbd0ea772547dd33d771509d917f8786754"}
Jul 24 16:47:25.799667 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.799647    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-4rmnz" event={"ID":"20548047-6c72-4740-bef0-b49e84fa8f37","Type":"ContainerStarted","Data":"063a426969a84023a567bbe25e7ee2f01c04e325dda2ec3f08afd750d2b85624"}
Jul 24 16:47:25.799750 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.799675    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-4rmnz" event={"ID":"20548047-6c72-4740-bef0-b49e84fa8f37","Type":"ContainerStarted","Data":"404871f565b654cb1f6daadfb985e7ff557b32ce730934385c4b1f15d29d4333"}
Jul 24 16:47:25.816454 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.816401    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-4rmnz" podStartSLOduration=0.8163818 podStartE2EDuration="816.3818ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:47:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:47:25.815317492 +0000 UTC m=+268.457281377" watchObservedRunningTime="2026-07-24 16:47:25.8163818 +0000 UTC m=+268.458345685"
Jul 24 16:47:25.830729 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.830664    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d185fb45-2628-4ccc-8afb-fa4ab0b344c6-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-wvdgz\" (UID: \"d185fb45-2628-4ccc-8afb-fa4ab0b344c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wvdgz"
Jul 24 16:47:25.830729 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.830702    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-service-ca-bundle\") pod \"router-default-746d84cf85-s4sxm\" (UID: \"f3934ff0-8fae-4d73-82e3-96276d9ff5b0\") " pod="openshift-ingress/router-default-746d84cf85-s4sxm"
Jul 24 16:47:25.830873 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:25.830736    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-metrics-certs\") pod \"router-default-746d84cf85-s4sxm\" (UID: \"f3934ff0-8fae-4d73-82e3-96276d9ff5b0\") " pod="openshift-ingress/router-default-746d84cf85-s4sxm"
Jul 24 16:47:25.830873 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:25.830825    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 24 16:47:25.830873 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:25.830855    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-service-ca-bundle podName:f3934ff0-8fae-4d73-82e3-96276d9ff5b0 nodeName:}" failed. No retries permitted until 2026-07-24 16:47:26.830834792 +0000 UTC m=+269.472798655 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-service-ca-bundle") pod "router-default-746d84cf85-s4sxm" (UID: "f3934ff0-8fae-4d73-82e3-96276d9ff5b0") : configmap references non-existent config key: service-ca.crt
Jul 24 16:47:25.831057 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:25.830880    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d185fb45-2628-4ccc-8afb-fa4ab0b344c6-samples-operator-tls podName:d185fb45-2628-4ccc-8afb-fa4ab0b344c6 nodeName:}" failed. No retries permitted until 2026-07-24 16:47:26.83087267 +0000 UTC m=+269.472836531 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d185fb45-2628-4ccc-8afb-fa4ab0b344c6-samples-operator-tls") pod "cluster-samples-operator-6748489656-wvdgz" (UID: "d185fb45-2628-4ccc-8afb-fa4ab0b344c6") : secret "samples-operator-tls" not found
Jul 24 16:47:25.831057 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:25.830920    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 24 16:47:25.831057 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:25.831004    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-metrics-certs podName:f3934ff0-8fae-4d73-82e3-96276d9ff5b0 nodeName:}" failed. No retries permitted until 2026-07-24 16:47:26.830987303 +0000 UTC m=+269.472951190 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-metrics-certs") pod "router-default-746d84cf85-s4sxm" (UID: "f3934ff0-8fae-4d73-82e3-96276d9ff5b0") : secret "router-metrics-certs-default" not found
Jul 24 16:47:26.841993 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:26.841953    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-metrics-certs\") pod \"router-default-746d84cf85-s4sxm\" (UID: \"f3934ff0-8fae-4d73-82e3-96276d9ff5b0\") " pod="openshift-ingress/router-default-746d84cf85-s4sxm"
Jul 24 16:47:26.842592 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:26.842123    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d185fb45-2628-4ccc-8afb-fa4ab0b344c6-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-wvdgz\" (UID: \"d185fb45-2628-4ccc-8afb-fa4ab0b344c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wvdgz"
Jul 24 16:47:26.842592 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:26.842150    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-service-ca-bundle\") pod \"router-default-746d84cf85-s4sxm\" (UID: \"f3934ff0-8fae-4d73-82e3-96276d9ff5b0\") " pod="openshift-ingress/router-default-746d84cf85-s4sxm"
Jul 24 16:47:26.842592 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:26.842294    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-service-ca-bundle podName:f3934ff0-8fae-4d73-82e3-96276d9ff5b0 nodeName:}" failed. No retries permitted until 2026-07-24 16:47:28.84227468 +0000 UTC m=+271.484238562 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-service-ca-bundle") pod "router-default-746d84cf85-s4sxm" (UID: "f3934ff0-8fae-4d73-82e3-96276d9ff5b0") : configmap references non-existent config key: service-ca.crt
Jul 24 16:47:26.842781 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:26.842707    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 24 16:47:26.842781 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:26.842754    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-metrics-certs podName:f3934ff0-8fae-4d73-82e3-96276d9ff5b0 nodeName:}" failed. No retries permitted until 2026-07-24 16:47:28.842739329 +0000 UTC m=+271.484703199 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-metrics-certs") pod "router-default-746d84cf85-s4sxm" (UID: "f3934ff0-8fae-4d73-82e3-96276d9ff5b0") : secret "router-metrics-certs-default" not found
Jul 24 16:47:26.842883 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:26.842808    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 24 16:47:26.842883 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:26.842840    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d185fb45-2628-4ccc-8afb-fa4ab0b344c6-samples-operator-tls podName:d185fb45-2628-4ccc-8afb-fa4ab0b344c6 nodeName:}" failed. No retries permitted until 2026-07-24 16:47:28.842829854 +0000 UTC m=+271.484793721 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d185fb45-2628-4ccc-8afb-fa4ab0b344c6-samples-operator-tls") pod "cluster-samples-operator-6748489656-wvdgz" (UID: "d185fb45-2628-4ccc-8afb-fa4ab0b344c6") : secret "samples-operator-tls" not found
Jul 24 16:47:28.861496 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:28.861472    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d185fb45-2628-4ccc-8afb-fa4ab0b344c6-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-wvdgz\" (UID: \"d185fb45-2628-4ccc-8afb-fa4ab0b344c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wvdgz"
Jul 24 16:47:28.861758 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:28.861505    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-service-ca-bundle\") pod \"router-default-746d84cf85-s4sxm\" (UID: \"f3934ff0-8fae-4d73-82e3-96276d9ff5b0\") " pod="openshift-ingress/router-default-746d84cf85-s4sxm"
Jul 24 16:47:28.861758 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:28.861527    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-metrics-certs\") pod \"router-default-746d84cf85-s4sxm\" (UID: \"f3934ff0-8fae-4d73-82e3-96276d9ff5b0\") " pod="openshift-ingress/router-default-746d84cf85-s4sxm"
Jul 24 16:47:28.861758 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:28.861625    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 24 16:47:28.861758 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:28.861650    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-service-ca-bundle podName:f3934ff0-8fae-4d73-82e3-96276d9ff5b0 nodeName:}" failed. No retries permitted until 2026-07-24 16:47:32.86163399 +0000 UTC m=+275.503597868 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-service-ca-bundle") pod "router-default-746d84cf85-s4sxm" (UID: "f3934ff0-8fae-4d73-82e3-96276d9ff5b0") : configmap references non-existent config key: service-ca.crt
Jul 24 16:47:28.861758 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:28.861631    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 24 16:47:28.861758 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:28.861689    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-metrics-certs podName:f3934ff0-8fae-4d73-82e3-96276d9ff5b0 nodeName:}" failed. No retries permitted until 2026-07-24 16:47:32.861677229 +0000 UTC m=+275.503641091 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-metrics-certs") pod "router-default-746d84cf85-s4sxm" (UID: "f3934ff0-8fae-4d73-82e3-96276d9ff5b0") : secret "router-metrics-certs-default" not found
Jul 24 16:47:28.861758 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:28.861704    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d185fb45-2628-4ccc-8afb-fa4ab0b344c6-samples-operator-tls podName:d185fb45-2628-4ccc-8afb-fa4ab0b344c6 nodeName:}" failed. No retries permitted until 2026-07-24 16:47:32.861698238 +0000 UTC m=+275.503662100 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d185fb45-2628-4ccc-8afb-fa4ab0b344c6-samples-operator-tls") pod "cluster-samples-operator-6748489656-wvdgz" (UID: "d185fb45-2628-4ccc-8afb-fa4ab0b344c6") : secret "samples-operator-tls" not found
Jul 24 16:47:29.155055 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:29.155017    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:29.599578 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:29.599491    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:29.700125 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:29.700092    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:29.814726 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:29.814690    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xl9vk" event={"ID":"e3460f62-6389-4673-9e33-e4e5aa0d4d9a","Type":"ContainerStarted","Data":"ea28021250bd740bbb86badbbce6cd94e71973c10cd33ee162f9ff9c52b61f70"}
Jul 24 16:47:29.816981 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:29.816958    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ddklx_fb5a9838-1e05-4f86-b46c-760bd7aaa038/console-operator/0.log"
Jul 24 16:47:29.817115 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:29.817004    2563 generic.go:356] "Generic (PLEG): container finished" podID="fb5a9838-1e05-4f86-b46c-760bd7aaa038" containerID="ffdd6a2b1e8697468e2880558f40946c1494e572ca0901c78e46bced623eaec6" exitCode=255
Jul 24 16:47:29.817115 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:29.817088    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-ddklx" event={"ID":"fb5a9838-1e05-4f86-b46c-760bd7aaa038","Type":"ContainerDied","Data":"ffdd6a2b1e8697468e2880558f40946c1494e572ca0901c78e46bced623eaec6"}
Jul 24 16:47:29.817292 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:29.817276    2563 scope.go:117] "RemoveContainer" containerID="ffdd6a2b1e8697468e2880558f40946c1494e572ca0901c78e46bced623eaec6"
Jul 24 16:47:29.818689 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:29.818658    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-5qj6l" event={"ID":"ef1872b5-49d6-4162-bcba-afea4c9ca621","Type":"ContainerStarted","Data":"6c491c3a8390c04e93470b442628b4f7f32ae67572266bd17c06b7087294192d"}
Jul 24 16:47:29.820218 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:29.820198    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-62xxb" event={"ID":"1f6e96ed-11c0-4dbb-afb0-231d599a870b","Type":"ContainerStarted","Data":"8b65e8031e61d8f57c03cf42c365e56c7e85360bedf7d25547211cacb2e6141c"}
Jul 24 16:47:29.838168 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:29.838115    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xl9vk" podStartSLOduration=1.5909467739999998 podStartE2EDuration="4.838098834s" totalImagesPullingTime="3.24715206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:47:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:47:29.836299453 +0000 UTC m=+272.478263338" watchObservedRunningTime="2026-07-24 16:47:29.838098834 +0000 UTC m=+272.480062718"
Jul 24 16:47:29.858507 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:29.858451    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-62xxb" podStartSLOduration=3.293012381 podStartE2EDuration="4.858435893s" totalImagesPullingTime="1.565423512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:47:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:47:29.857858153 +0000 UTC m=+272.499822037" watchObservedRunningTime="2026-07-24 16:47:29.858435893 +0000 UTC m=+272.500399778"
Jul 24 16:47:29.868076 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:29.868051    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:29.896174 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:29.896121    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-5qj6l" podStartSLOduration=1.6643496660000001 podStartE2EDuration="4.896099093s" totalImagesPullingTime="3.231749427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:47:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:47:29.894093897 +0000 UTC m=+272.536057783" watchObservedRunningTime="2026-07-24 16:47:29.896099093 +0000 UTC m=+272.538062981"
Jul 24 16:47:29.983481 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:29.983430    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:29.992481 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:29.992455    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-j7rx8"]
Jul 24 16:47:29.995554 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:29.995534    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-j7rx8"
Jul 24 16:47:29.999179 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:29.999158    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 24 16:47:29.999311 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:29.999252    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 24 16:47:29.999311 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:29.999282    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k88vr\""
Jul 24 16:47:30.016811 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:30.016775    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-j7rx8"]
Jul 24 16:47:30.065472 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:30.065437    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:30.073539 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:30.073513    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c99ab05-563d-4bc8-882a-49defb5b97a9-data-volume\") pod \"insights-runtime-extractor-j7rx8\" (UID: \"3c99ab05-563d-4bc8-882a-49defb5b97a9\") " pod="openshift-insights/insights-runtime-extractor-j7rx8"
Jul 24 16:47:30.073673 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:30.073592    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c99ab05-563d-4bc8-882a-49defb5b97a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7rx8\" (UID: \"3c99ab05-563d-4bc8-882a-49defb5b97a9\") " pod="openshift-insights/insights-runtime-extractor-j7rx8"
Jul 24 16:47:30.073673 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:30.073609    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3c99ab05-563d-4bc8-882a-49defb5b97a9-crio-socket\") pod \"insights-runtime-extractor-j7rx8\" (UID: \"3c99ab05-563d-4bc8-882a-49defb5b97a9\") " pod="openshift-insights/insights-runtime-extractor-j7rx8"
Jul 24 16:47:30.073673 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:30.073637    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c99ab05-563d-4bc8-882a-49defb5b97a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7rx8\" (UID: \"3c99ab05-563d-4bc8-882a-49defb5b97a9\") " pod="openshift-insights/insights-runtime-extractor-j7rx8"
Jul 24 16:47:30.073673 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:30.073653    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bpm5\" (UniqueName: \"kubernetes.io/projected/3c99ab05-563d-4bc8-882a-49defb5b97a9-kube-api-access-5bpm5\") pod \"insights-runtime-extractor-j7rx8\" (UID: \"3c99ab05-563d-4bc8-882a-49defb5b97a9\") " pod="openshift-insights/insights-runtime-extractor-j7rx8"
Jul 24 16:47:30.175183 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:30.175144    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c99ab05-563d-4bc8-882a-49defb5b97a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7rx8\" (UID: \"3c99ab05-563d-4bc8-882a-49defb5b97a9\") " pod="openshift-insights/insights-runtime-extractor-j7rx8"
Jul 24 16:47:30.175359 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:30.175195    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3c99ab05-563d-4bc8-882a-49defb5b97a9-crio-socket\") pod \"insights-runtime-extractor-j7rx8\" (UID: \"3c99ab05-563d-4bc8-882a-49defb5b97a9\") " pod="openshift-insights/insights-runtime-extractor-j7rx8"
Jul 24 16:47:30.175359 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:30.175245    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c99ab05-563d-4bc8-882a-49defb5b97a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7rx8\" (UID: \"3c99ab05-563d-4bc8-882a-49defb5b97a9\") " pod="openshift-insights/insights-runtime-extractor-j7rx8"
Jul 24 16:47:30.175359 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:30.175276    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 16:47:30.175359 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:30.175271    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bpm5\" (UniqueName: \"kubernetes.io/projected/3c99ab05-563d-4bc8-882a-49defb5b97a9-kube-api-access-5bpm5\") pod \"insights-runtime-extractor-j7rx8\" (UID: \"3c99ab05-563d-4bc8-882a-49defb5b97a9\") " pod="openshift-insights/insights-runtime-extractor-j7rx8"
Jul 24 16:47:30.175359 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:30.175353    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c99ab05-563d-4bc8-882a-49defb5b97a9-kube-rbac-proxy-cm podName:3c99ab05-563d-4bc8-882a-49defb5b97a9 nodeName:}" failed. No retries permitted until 2026-07-24 16:47:30.675332886 +0000 UTC m=+273.317296766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3c99ab05-563d-4bc8-882a-49defb5b97a9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j7rx8" (UID: "3c99ab05-563d-4bc8-882a-49defb5b97a9") : configmap "kube-rbac-proxy" not found
Jul 24 16:47:30.175657 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:30.175418    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c99ab05-563d-4bc8-882a-49defb5b97a9-data-volume\") pod \"insights-runtime-extractor-j7rx8\" (UID: \"3c99ab05-563d-4bc8-882a-49defb5b97a9\") " pod="openshift-insights/insights-runtime-extractor-j7rx8"
Jul 24 16:47:30.175657 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:30.175522    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3c99ab05-563d-4bc8-882a-49defb5b97a9-crio-socket\") pod \"insights-runtime-extractor-j7rx8\" (UID: \"3c99ab05-563d-4bc8-882a-49defb5b97a9\") " pod="openshift-insights/insights-runtime-extractor-j7rx8"
Jul 24 16:47:30.175657 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:30.175627    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 16:47:30.175809 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:30.175687    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c99ab05-563d-4bc8-882a-49defb5b97a9-insights-runtime-extractor-tls podName:3c99ab05-563d-4bc8-882a-49defb5b97a9 nodeName:}" failed. No retries permitted until 2026-07-24 16:47:30.675670927 +0000 UTC m=+273.317634793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3c99ab05-563d-4bc8-882a-49defb5b97a9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j7rx8" (UID: "3c99ab05-563d-4bc8-882a-49defb5b97a9") : secret "insights-runtime-extractor-tls" not found
Jul 24 16:47:30.175809 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:30.175801    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c99ab05-563d-4bc8-882a-49defb5b97a9-data-volume\") pod \"insights-runtime-extractor-j7rx8\" (UID: \"3c99ab05-563d-4bc8-882a-49defb5b97a9\") " pod="openshift-insights/insights-runtime-extractor-j7rx8"
Jul 24 16:47:30.186456 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:30.186430    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bpm5\" (UniqueName: \"kubernetes.io/projected/3c99ab05-563d-4bc8-882a-49defb5b97a9-kube-api-access-5bpm5\") pod \"insights-runtime-extractor-j7rx8\" (UID: \"3c99ab05-563d-4bc8-882a-49defb5b97a9\") " pod="openshift-insights/insights-runtime-extractor-j7rx8"
Jul 24 16:47:30.227787 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:30.227750    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:30.294296 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:30.294265    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:30.348870 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:30.348842    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:30.679180 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:30.679141    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c99ab05-563d-4bc8-882a-49defb5b97a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7rx8\" (UID: \"3c99ab05-563d-4bc8-882a-49defb5b97a9\") " pod="openshift-insights/insights-runtime-extractor-j7rx8"
Jul 24 16:47:30.679331 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:30.679275    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c99ab05-563d-4bc8-882a-49defb5b97a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7rx8\" (UID: \"3c99ab05-563d-4bc8-882a-49defb5b97a9\") " pod="openshift-insights/insights-runtime-extractor-j7rx8"
Jul 24 16:47:30.679331 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:30.679298    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 16:47:30.679420 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:30.679363    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c99ab05-563d-4bc8-882a-49defb5b97a9-insights-runtime-extractor-tls podName:3c99ab05-563d-4bc8-882a-49defb5b97a9 nodeName:}" failed. No retries permitted until 2026-07-24 16:47:31.679348292 +0000 UTC m=+274.321312153 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3c99ab05-563d-4bc8-882a-49defb5b97a9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j7rx8" (UID: "3c99ab05-563d-4bc8-882a-49defb5b97a9") : secret "insights-runtime-extractor-tls" not found
Jul 24 16:47:30.679420 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:30.679367    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 16:47:30.679420 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:30.679411    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c99ab05-563d-4bc8-882a-49defb5b97a9-kube-rbac-proxy-cm podName:3c99ab05-563d-4bc8-882a-49defb5b97a9 nodeName:}" failed. No retries permitted until 2026-07-24 16:47:31.679396401 +0000 UTC m=+274.321360262 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3c99ab05-563d-4bc8-882a-49defb5b97a9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j7rx8" (UID: "3c99ab05-563d-4bc8-882a-49defb5b97a9") : configmap "kube-rbac-proxy" not found
Jul 24 16:47:30.825059 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:30.825031    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ddklx_fb5a9838-1e05-4f86-b46c-760bd7aaa038/console-operator/1.log"
Jul 24 16:47:30.825449 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:30.825435    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ddklx_fb5a9838-1e05-4f86-b46c-760bd7aaa038/console-operator/0.log"
Jul 24 16:47:30.825504 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:30.825468    2563 generic.go:356] "Generic (PLEG): container finished" podID="fb5a9838-1e05-4f86-b46c-760bd7aaa038" containerID="c6cd1e659014482553a9a18803c199d1e60534f0524df5082c80b3af69b13f09" exitCode=255
Jul 24 16:47:30.825572 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:30.825552    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-ddklx" event={"ID":"fb5a9838-1e05-4f86-b46c-760bd7aaa038","Type":"ContainerDied","Data":"c6cd1e659014482553a9a18803c199d1e60534f0524df5082c80b3af69b13f09"}
Jul 24 16:47:30.825616 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:30.825606    2563 scope.go:117] "RemoveContainer" containerID="ffdd6a2b1e8697468e2880558f40946c1494e572ca0901c78e46bced623eaec6"
Jul 24 16:47:30.825856 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:30.825831    2563 scope.go:117] "RemoveContainer" containerID="c6cd1e659014482553a9a18803c199d1e60534f0524df5082c80b3af69b13f09"
Jul 24 16:47:30.826027 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:30.826009    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-ddklx_openshift-console-operator(fb5a9838-1e05-4f86-b46c-760bd7aaa038)\"" pod="openshift-console-operator/console-operator-584dcb6b54-ddklx" podUID="fb5a9838-1e05-4f86-b46c-760bd7aaa038"
Jul 24 16:47:31.496042 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:31.496012    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:31.554464 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:31.554432    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:31.689136 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:31.689092    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c99ab05-563d-4bc8-882a-49defb5b97a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7rx8\" (UID: \"3c99ab05-563d-4bc8-882a-49defb5b97a9\") " pod="openshift-insights/insights-runtime-extractor-j7rx8"
Jul 24 16:47:31.689329 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:31.689224    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c99ab05-563d-4bc8-882a-49defb5b97a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7rx8\" (UID: \"3c99ab05-563d-4bc8-882a-49defb5b97a9\") " pod="openshift-insights/insights-runtime-extractor-j7rx8"
Jul 24 16:47:31.689329 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:31.689267    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 16:47:31.689329 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:31.689293    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 16:47:31.689463 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:31.689358    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c99ab05-563d-4bc8-882a-49defb5b97a9-kube-rbac-proxy-cm podName:3c99ab05-563d-4bc8-882a-49defb5b97a9 nodeName:}" failed. No retries permitted until 2026-07-24 16:47:33.68933916 +0000 UTC m=+276.331303024 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3c99ab05-563d-4bc8-882a-49defb5b97a9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j7rx8" (UID: "3c99ab05-563d-4bc8-882a-49defb5b97a9") : configmap "kube-rbac-proxy" not found
Jul 24 16:47:31.689463 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:31.689371    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c99ab05-563d-4bc8-882a-49defb5b97a9-insights-runtime-extractor-tls podName:3c99ab05-563d-4bc8-882a-49defb5b97a9 nodeName:}" failed. No retries permitted until 2026-07-24 16:47:33.689365004 +0000 UTC m=+276.331328866 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3c99ab05-563d-4bc8-882a-49defb5b97a9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j7rx8" (UID: "3c99ab05-563d-4bc8-882a-49defb5b97a9") : secret "insights-runtime-extractor-tls" not found
Jul 24 16:47:31.830715 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:31.830647    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ddklx_fb5a9838-1e05-4f86-b46c-760bd7aaa038/console-operator/1.log"
Jul 24 16:47:31.831015 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:31.830997    2563 scope.go:117] "RemoveContainer" containerID="c6cd1e659014482553a9a18803c199d1e60534f0524df5082c80b3af69b13f09"
Jul 24 16:47:31.831166 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:31.831146    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-ddklx_openshift-console-operator(fb5a9838-1e05-4f86-b46c-760bd7aaa038)\"" pod="openshift-console-operator/console-operator-584dcb6b54-ddklx" podUID="fb5a9838-1e05-4f86-b46c-760bd7aaa038"
Jul 24 16:47:31.971005 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:31.970979    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g5sl6_f263630d-88c9-4b5d-8afb-7075a1118555/dns-node-resolver/0.log"
Jul 24 16:47:32.357246 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:32.357212    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:32.900115 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:32.900085    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d185fb45-2628-4ccc-8afb-fa4ab0b344c6-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-wvdgz\" (UID: \"d185fb45-2628-4ccc-8afb-fa4ab0b344c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wvdgz"
Jul 24 16:47:32.900115 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:32.900116    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-service-ca-bundle\") pod \"router-default-746d84cf85-s4sxm\" (UID: \"f3934ff0-8fae-4d73-82e3-96276d9ff5b0\") " pod="openshift-ingress/router-default-746d84cf85-s4sxm"
Jul 24 16:47:32.900623 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:32.900143    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-metrics-certs\") pod \"router-default-746d84cf85-s4sxm\" (UID: \"f3934ff0-8fae-4d73-82e3-96276d9ff5b0\") " pod="openshift-ingress/router-default-746d84cf85-s4sxm"
Jul 24 16:47:32.900623 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:32.900234    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 24 16:47:32.900623 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:32.900234    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 24 16:47:32.900623 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:32.900276    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-service-ca-bundle podName:f3934ff0-8fae-4d73-82e3-96276d9ff5b0 nodeName:}" failed. No retries permitted until 2026-07-24 16:47:40.900254271 +0000 UTC m=+283.542218132 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-service-ca-bundle") pod "router-default-746d84cf85-s4sxm" (UID: "f3934ff0-8fae-4d73-82e3-96276d9ff5b0") : configmap references non-existent config key: service-ca.crt
Jul 24 16:47:32.900623 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:32.900311    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d185fb45-2628-4ccc-8afb-fa4ab0b344c6-samples-operator-tls podName:d185fb45-2628-4ccc-8afb-fa4ab0b344c6 nodeName:}" failed. No retries permitted until 2026-07-24 16:47:40.900299368 +0000 UTC m=+283.542263233 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d185fb45-2628-4ccc-8afb-fa4ab0b344c6-samples-operator-tls") pod "cluster-samples-operator-6748489656-wvdgz" (UID: "d185fb45-2628-4ccc-8afb-fa4ab0b344c6") : secret "samples-operator-tls" not found
Jul 24 16:47:32.900623 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:32.900345    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-metrics-certs podName:f3934ff0-8fae-4d73-82e3-96276d9ff5b0 nodeName:}" failed. No retries permitted until 2026-07-24 16:47:40.900326533 +0000 UTC m=+283.542290398 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-metrics-certs") pod "router-default-746d84cf85-s4sxm" (UID: "f3934ff0-8fae-4d73-82e3-96276d9ff5b0") : secret "router-metrics-certs-default" not found
Jul 24 16:47:33.170299 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:33.170230    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vq4cx_ce9a1972-6a1e-4e0d-9dab-0e25820dedf1/node-ca/0.log"
Jul 24 16:47:33.309121 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:33.309091    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:33.449421 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:33.449348    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:33.547797 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:33.547765    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:33.663862 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:33.663830    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:33.707208 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:33.707155    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c99ab05-563d-4bc8-882a-49defb5b97a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7rx8\" (UID: \"3c99ab05-563d-4bc8-882a-49defb5b97a9\") " pod="openshift-insights/insights-runtime-extractor-j7rx8"
Jul 24 16:47:33.707208 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:33.707195    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c99ab05-563d-4bc8-882a-49defb5b97a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7rx8\" (UID: \"3c99ab05-563d-4bc8-882a-49defb5b97a9\") " pod="openshift-insights/insights-runtime-extractor-j7rx8"
Jul 24 16:47:33.707330 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:33.707268    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 16:47:33.707330 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:33.707324    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 16:47:33.707391 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:33.707338    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c99ab05-563d-4bc8-882a-49defb5b97a9-kube-rbac-proxy-cm podName:3c99ab05-563d-4bc8-882a-49defb5b97a9 nodeName:}" failed. No retries permitted until 2026-07-24 16:47:37.707323608 +0000 UTC m=+280.349287471 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3c99ab05-563d-4bc8-882a-49defb5b97a9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j7rx8" (UID: "3c99ab05-563d-4bc8-882a-49defb5b97a9") : configmap "kube-rbac-proxy" not found
Jul 24 16:47:33.707391 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:33.707361    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c99ab05-563d-4bc8-882a-49defb5b97a9-insights-runtime-extractor-tls podName:3c99ab05-563d-4bc8-882a-49defb5b97a9 nodeName:}" failed. No retries permitted until 2026-07-24 16:47:37.707351317 +0000 UTC m=+280.349315179 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3c99ab05-563d-4bc8-882a-49defb5b97a9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j7rx8" (UID: "3c99ab05-563d-4bc8-882a-49defb5b97a9") : secret "insights-runtime-extractor-tls" not found
Jul 24 16:47:33.763610 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:33.763579    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:33.770320 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:33.770294    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8zdrn_17291b08-516e-48fe-9b2a-e8db21ed7c6c/serve-healthcheck-canary/0.log"
Jul 24 16:47:34.201723 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:34.201693    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:34.461872 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:34.461800    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:34.995314 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:34.995283    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:35.393342 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:35.393315    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-ddklx"
Jul 24 16:47:35.393686 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:35.393367    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-ddklx"
Jul 24 16:47:35.393686 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:35.393658    2563 scope.go:117] "RemoveContainer" containerID="c6cd1e659014482553a9a18803c199d1e60534f0524df5082c80b3af69b13f09"
Jul 24 16:47:35.393813 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:35.393795    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-ddklx_openshift-console-operator(fb5a9838-1e05-4f86-b46c-760bd7aaa038)\"" pod="openshift-console-operator/console-operator-584dcb6b54-ddklx" podUID="fb5a9838-1e05-4f86-b46c-760bd7aaa038"
Jul 24 16:47:35.396951 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:35.396922    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:35.598813 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:35.598783    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:35.794872 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:35.794784    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:36.002862 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:36.002814    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:37.027227 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:37.027192    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:37.190952 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:37.190902    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:37.401979 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:37.401950    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:37.502356 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:37.502325    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:37.603128 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:37.603076    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:37.742949 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:37.742856    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c99ab05-563d-4bc8-882a-49defb5b97a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7rx8\" (UID: \"3c99ab05-563d-4bc8-882a-49defb5b97a9\") " pod="openshift-insights/insights-runtime-extractor-j7rx8"
Jul 24 16:47:37.743083 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:37.743024    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c99ab05-563d-4bc8-882a-49defb5b97a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7rx8\" (UID: \"3c99ab05-563d-4bc8-882a-49defb5b97a9\") " pod="openshift-insights/insights-runtime-extractor-j7rx8"
Jul 24 16:47:37.743083 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:37.743035    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 16:47:37.743153 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:37.743102    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 16:47:37.743153 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:37.743114    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c99ab05-563d-4bc8-882a-49defb5b97a9-insights-runtime-extractor-tls podName:3c99ab05-563d-4bc8-882a-49defb5b97a9 nodeName:}" failed. No retries permitted until 2026-07-24 16:47:45.743095651 +0000 UTC m=+288.385059525 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3c99ab05-563d-4bc8-882a-49defb5b97a9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j7rx8" (UID: "3c99ab05-563d-4bc8-882a-49defb5b97a9") : secret "insights-runtime-extractor-tls" not found
Jul 24 16:47:37.743153 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:37.743136    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c99ab05-563d-4bc8-882a-49defb5b97a9-kube-rbac-proxy-cm podName:3c99ab05-563d-4bc8-882a-49defb5b97a9 nodeName:}" failed. No retries permitted until 2026-07-24 16:47:45.743126332 +0000 UTC m=+288.385090210 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3c99ab05-563d-4bc8-882a-49defb5b97a9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j7rx8" (UID: "3c99ab05-563d-4bc8-882a-49defb5b97a9") : configmap "kube-rbac-proxy" not found
Jul 24 16:47:37.803420 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:37.803383    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:38.197354 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:38.197322    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:38.542856 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:38.542776    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb" podUID="6e7f3a6f-1da1-4a07-b736-512dfb2afd6a"
Jul 24 16:47:38.542856 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:38.542775    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-74c9f884b7-mt8vp" podUID="7960f9f4-815e-4b60-bd8c-2a0654a61ae6"
Jul 24 16:47:38.542856 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:38.542774    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-vrscm" podUID="708abf59-5c2a-46e2-8506-386e91731078"
Jul 24 16:47:38.850979 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:38.850886    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vrscm"
Jul 24 16:47:38.850979 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:38.850925    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-mt8vp"
Jul 24 16:47:38.851159 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:38.851047    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:47:39.011787 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:39.011758    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:39.191702 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:39.191668    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:39.398792 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:39.398754    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:39.793743 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:39.793698    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:39.848721 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:39.848696    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:40.606867 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:40.606840    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:40.969719 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:40.969684    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d185fb45-2628-4ccc-8afb-fa4ab0b344c6-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-wvdgz\" (UID: \"d185fb45-2628-4ccc-8afb-fa4ab0b344c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wvdgz"
Jul 24 16:47:40.969719 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:40.969717    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-service-ca-bundle\") pod \"router-default-746d84cf85-s4sxm\" (UID: \"f3934ff0-8fae-4d73-82e3-96276d9ff5b0\") " pod="openshift-ingress/router-default-746d84cf85-s4sxm"
Jul 24 16:47:40.969930 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:40.969743    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-metrics-certs\") pod \"router-default-746d84cf85-s4sxm\" (UID: \"f3934ff0-8fae-4d73-82e3-96276d9ff5b0\") " pod="openshift-ingress/router-default-746d84cf85-s4sxm"
Jul 24 16:47:40.970452 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:40.970428    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-service-ca-bundle\") pod \"router-default-746d84cf85-s4sxm\" (UID: \"f3934ff0-8fae-4d73-82e3-96276d9ff5b0\") " pod="openshift-ingress/router-default-746d84cf85-s4sxm"
Jul 24 16:47:40.972149 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:40.972127    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d185fb45-2628-4ccc-8afb-fa4ab0b344c6-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-wvdgz\" (UID: \"d185fb45-2628-4ccc-8afb-fa4ab0b344c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wvdgz"
Jul 24 16:47:40.972679 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:40.972655    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3934ff0-8fae-4d73-82e3-96276d9ff5b0-metrics-certs\") pod \"router-default-746d84cf85-s4sxm\" (UID: \"f3934ff0-8fae-4d73-82e3-96276d9ff5b0\") " pod="openshift-ingress/router-default-746d84cf85-s4sxm"
Jul 24 16:47:41.014915 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:41.014887    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:41.094185 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:41.094154    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-746d84cf85-s4sxm"
Jul 24 16:47:41.099433 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:41.099411    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wvdgz"
Jul 24 16:47:41.196909 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:41.196875    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:41.230924 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:41.230819    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wvdgz"]
Jul 24 16:47:41.251412 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:41.251387    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-746d84cf85-s4sxm"]
Jul 24 16:47:41.254099 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:47:41.254075    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3934ff0_8fae_4d73_82e3_96276d9ff5b0.slice/crio-6e2752c255f86fe018827338ae9af6e8f8a4bdc289b469966a26fa715ce27fda WatchSource:0}: Error finding container 6e2752c255f86fe018827338ae9af6e8f8a4bdc289b469966a26fa715ce27fda: Status 404 returned error can't find the container with id 6e2752c255f86fe018827338ae9af6e8f8a4bdc289b469966a26fa715ce27fda
Jul 24 16:47:41.394486 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:41.394404    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:41.596436 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:41.596334    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:41.860186 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:41.860093    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-746d84cf85-s4sxm" event={"ID":"f3934ff0-8fae-4d73-82e3-96276d9ff5b0","Type":"ContainerStarted","Data":"4ae752e708e4f16efba29195ae2249875f4ef01a90b96c99003ddf71916b50d1"}
Jul 24 16:47:41.860186 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:41.860157    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-746d84cf85-s4sxm" event={"ID":"f3934ff0-8fae-4d73-82e3-96276d9ff5b0","Type":"ContainerStarted","Data":"6e2752c255f86fe018827338ae9af6e8f8a4bdc289b469966a26fa715ce27fda"}
Jul 24 16:47:41.861126 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:41.861108    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wvdgz" event={"ID":"d185fb45-2628-4ccc-8afb-fa4ab0b344c6","Type":"ContainerStarted","Data":"198212d53f5b45e25e780c5d2ea7d4a97cd5894fa236fc8bceebb30dd49a2ed3"}
Jul 24 16:47:41.878720 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:41.878682    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-746d84cf85-s4sxm" podStartSLOduration=16.878670367 podStartE2EDuration="16.878670367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:47:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:47:41.877135591 +0000 UTC m=+284.519099488" watchObservedRunningTime="2026-07-24 16:47:41.878670367 +0000 UTC m=+284.520634251"
Jul 24 16:47:42.094663 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:42.094626    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-746d84cf85-s4sxm"
Jul 24 16:47:42.097545 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:42.097521    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-746d84cf85-s4sxm"
Jul 24 16:47:42.281738 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:42.281703    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-registry-tls\") pod \"image-registry-c8f78f7cf-2sdpb\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") " pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:47:42.281893 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:42.281747    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/708abf59-5c2a-46e2-8506-386e91731078-metrics-tls\") pod \"dns-default-vrscm\" (UID: \"708abf59-5c2a-46e2-8506-386e91731078\") " pod="openshift-dns/dns-default-vrscm"
Jul 24 16:47:42.281893 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:42.281795    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7960f9f4-815e-4b60-bd8c-2a0654a61ae6-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-mt8vp\" (UID: \"7960f9f4-815e-4b60-bd8c-2a0654a61ae6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-mt8vp"
Jul 24 16:47:42.284722 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:42.284691    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/708abf59-5c2a-46e2-8506-386e91731078-metrics-tls\") pod \"dns-default-vrscm\" (UID: \"708abf59-5c2a-46e2-8506-386e91731078\") " pod="openshift-dns/dns-default-vrscm"
Jul 24 16:47:42.284843 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:42.284812    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-registry-tls\") pod \"image-registry-c8f78f7cf-2sdpb\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") " pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:47:42.284843 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:42.284817    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7960f9f4-815e-4b60-bd8c-2a0654a61ae6-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-mt8vp\" (UID: \"7960f9f4-815e-4b60-bd8c-2a0654a61ae6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-mt8vp"
Jul 24 16:47:42.454076 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:42.454042    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4mtk4\""
Jul 24 16:47:42.454251 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:42.454108    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-rdbwx\""
Jul 24 16:47:42.454251 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:42.454108    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6r7g7\""
Jul 24 16:47:42.462037 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:42.462005    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vrscm"
Jul 24 16:47:42.462147 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:42.462131    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:47:42.462249 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:42.462162    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-mt8vp"
Jul 24 16:47:42.618682 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:42.618642    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vrscm"]
Jul 24 16:47:42.621269 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:47:42.621240    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod708abf59_5c2a_46e2_8506_386e91731078.slice/crio-8da95ccc867e8b6c2d4543b4751b09f4dd14636aed14a8002ca25712e455a34b WatchSource:0}: Error finding container 8da95ccc867e8b6c2d4543b4751b09f4dd14636aed14a8002ca25712e455a34b: Status 404 returned error can't find the container with id 8da95ccc867e8b6c2d4543b4751b09f4dd14636aed14a8002ca25712e455a34b
Jul 24 16:47:42.800600 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:42.800497    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:42.842776 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:42.842743    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-mt8vp"]
Jul 24 16:47:42.846010 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:42.845987    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c8f78f7cf-2sdpb"]
Jul 24 16:47:42.852269 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:42.852245    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:42.865710 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:42.865686    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vrscm" event={"ID":"708abf59-5c2a-46e2-8506-386e91731078","Type":"ContainerStarted","Data":"8da95ccc867e8b6c2d4543b4751b09f4dd14636aed14a8002ca25712e455a34b"}
Jul 24 16:47:42.866096 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:42.865883    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-746d84cf85-s4sxm"
Jul 24 16:47:42.867189 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:42.867173    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-746d84cf85-s4sxm"
Jul 24 16:47:42.997588 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:42.997559    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:43.195825 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:43.195798    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:43.214671 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:47:43.214642    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7960f9f4_815e_4b60_bd8c_2a0654a61ae6.slice/crio-91ae4d98c598e79148c54bf1856339f8e28e5d7dea7eeacf6713044155b12024 WatchSource:0}: Error finding container 91ae4d98c598e79148c54bf1856339f8e28e5d7dea7eeacf6713044155b12024: Status 404 returned error can't find the container with id 91ae4d98c598e79148c54bf1856339f8e28e5d7dea7eeacf6713044155b12024
Jul 24 16:47:43.215083 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:47:43.215063    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e7f3a6f_1da1_4a07_b736_512dfb2afd6a.slice/crio-e63dd6e8320908a68c4598e679b1854b479a77c1f98840c8f4c0b897f53d992b WatchSource:0}: Error finding container e63dd6e8320908a68c4598e679b1854b479a77c1f98840c8f4c0b897f53d992b: Status 404 returned error can't find the container with id e63dd6e8320908a68c4598e679b1854b479a77c1f98840c8f4c0b897f53d992b
Jul 24 16:47:43.409407 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:43.409379    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:43.507130 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:43.507041    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:43.606096 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:43.606060    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:43.709372 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:43.709338    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:43.872496 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:43.872403    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wvdgz" event={"ID":"d185fb45-2628-4ccc-8afb-fa4ab0b344c6","Type":"ContainerStarted","Data":"1963e92527e7b51faa8824589b1495e0b8bb6a29d066fc69b122d8d39ddfc31f"}
Jul 24 16:47:43.872496 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:43.872473    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wvdgz" event={"ID":"d185fb45-2628-4ccc-8afb-fa4ab0b344c6","Type":"ContainerStarted","Data":"402535ab1e93f083136bec9afc40dc050dd0deafc067a0b86d8e7964f72a0a2f"}
Jul 24 16:47:43.873893 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:43.873866    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-mt8vp" event={"ID":"7960f9f4-815e-4b60-bd8c-2a0654a61ae6","Type":"ContainerStarted","Data":"91ae4d98c598e79148c54bf1856339f8e28e5d7dea7eeacf6713044155b12024"}
Jul 24 16:47:43.875552 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:43.875526    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb" event={"ID":"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a","Type":"ContainerStarted","Data":"17a2c0edffad5621dae52f44d4069a52dc0d5881865027c4120a92e1dfbfb2ae"}
Jul 24 16:47:43.875656 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:43.875569    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb" event={"ID":"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a","Type":"ContainerStarted","Data":"e63dd6e8320908a68c4598e679b1854b479a77c1f98840c8f4c0b897f53d992b"}
Jul 24 16:47:43.875866 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:43.875840    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:47:43.889845 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:43.889656    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wvdgz" podStartSLOduration=16.8953457 podStartE2EDuration="18.889637784s" totalImagesPullingTime="1.994292084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:47:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:47:43.888746082 +0000 UTC m=+286.530709967" watchObservedRunningTime="2026-07-24 16:47:43.889637784 +0000 UTC m=+286.531601671"
Jul 24 16:47:43.908597 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:43.908558    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb" podStartSLOduration=284.908540334 podStartE2EDuration="4m44.908540334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:42:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:47:43.907725327 +0000 UTC m=+286.549689237" watchObservedRunningTime="2026-07-24 16:47:43.908540334 +0000 UTC m=+286.550504223"
Jul 24 16:47:43.999117 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:43.999075    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:44.204959 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:44.204912    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:44.305879 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:44.305845    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:44.363657 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:44.363614    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:44.417704 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:44.417663    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:45.812685 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:45.812648    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c99ab05-563d-4bc8-882a-49defb5b97a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7rx8\" (UID: \"3c99ab05-563d-4bc8-882a-49defb5b97a9\") " pod="openshift-insights/insights-runtime-extractor-j7rx8"
Jul 24 16:47:45.813057 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:45.812699    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c99ab05-563d-4bc8-882a-49defb5b97a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7rx8\" (UID: \"3c99ab05-563d-4bc8-882a-49defb5b97a9\") " pod="openshift-insights/insights-runtime-extractor-j7rx8"
Jul 24 16:47:45.813199 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:45.813180    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c99ab05-563d-4bc8-882a-49defb5b97a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7rx8\" (UID: \"3c99ab05-563d-4bc8-882a-49defb5b97a9\") " pod="openshift-insights/insights-runtime-extractor-j7rx8"
Jul 24 16:47:45.814919 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:45.814902    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c99ab05-563d-4bc8-882a-49defb5b97a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7rx8\" (UID: \"3c99ab05-563d-4bc8-882a-49defb5b97a9\") " pod="openshift-insights/insights-runtime-extractor-j7rx8"
Jul 24 16:47:45.886254 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:45.886224    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vrscm" event={"ID":"708abf59-5c2a-46e2-8506-386e91731078","Type":"ContainerStarted","Data":"8f7cd03b4494385923ce8e84e9caff0f69edc7135bb96f5a8f951e0e91234f63"}
Jul 24 16:47:45.886408 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:45.886313    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vrscm" event={"ID":"708abf59-5c2a-46e2-8506-386e91731078","Type":"ContainerStarted","Data":"14d66b39efd716d20c049993f2d63f6fd8249fefc382b6b1d220d5124d471b4a"}
Jul 24 16:47:45.886408 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:45.886353    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vrscm"
Jul 24 16:47:45.887717 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:45.887694    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-mt8vp" event={"ID":"7960f9f4-815e-4b60-bd8c-2a0654a61ae6","Type":"ContainerStarted","Data":"f783fe9e18950d0b8c7f4254af3f20f11324f0b9c8bc029a9cb51ef8f4c77009"}
Jul 24 16:47:45.903263 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:45.903223    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vrscm" podStartSLOduration=251.556157596 podStartE2EDuration="4m13.903201912s" totalImagesPullingTime="2.347044316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:43:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:47:45.902147765 +0000 UTC m=+288.544111650" watchObservedRunningTime="2026-07-24 16:47:45.903201912 +0000 UTC m=+288.545165796"
Jul 24 16:47:45.906743 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:45.906726    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-j7rx8"
Jul 24 16:47:45.917517 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:45.917481    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-mt8vp" podStartSLOduration=266.16844587 podStartE2EDuration="4m27.917467775s" totalImagesPullingTime="1.749021905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:43:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:47:45.916190678 +0000 UTC m=+288.558154562" watchObservedRunningTime="2026-07-24 16:47:45.917467775 +0000 UTC m=+288.559431697"
Jul 24 16:47:46.025914 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:46.025873    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-j7rx8"]
Jul 24 16:47:46.028194 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:47:46.028167    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c99ab05_563d_4bc8_882a_49defb5b97a9.slice/crio-22f325cd87abb037754d87c6e8ee3f30631784c59b8dbb2813064d87902f5981 WatchSource:0}: Error finding container 22f325cd87abb037754d87c6e8ee3f30631784c59b8dbb2813064d87902f5981: Status 404 returned error can't find the container with id 22f325cd87abb037754d87c6e8ee3f30631784c59b8dbb2813064d87902f5981
Jul 24 16:47:46.892843 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:46.892801    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j7rx8" event={"ID":"3c99ab05-563d-4bc8-882a-49defb5b97a9","Type":"ContainerStarted","Data":"22f325cd87abb037754d87c6e8ee3f30631784c59b8dbb2813064d87902f5981"}
Jul 24 16:47:47.999064 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:47.999030    2563 scope.go:117] "RemoveContainer" containerID="c6cd1e659014482553a9a18803c199d1e60534f0524df5082c80b3af69b13f09"
Jul 24 16:47:48.902238 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:48.902202    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ddklx_fb5a9838-1e05-4f86-b46c-760bd7aaa038/console-operator/1.log"
Jul 24 16:47:48.902437 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:48.902317    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-ddklx" event={"ID":"fb5a9838-1e05-4f86-b46c-760bd7aaa038","Type":"ContainerStarted","Data":"9ae34d6b5e5b7abed389129dddb5d21ad744e400ae7827afac78edddff68a614"}
Jul 24 16:47:48.902671 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:48.902645    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-ddklx"
Jul 24 16:47:48.918596 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:48.918545    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-ddklx" podStartSLOduration=20.79273864 podStartE2EDuration="23.918528501s" totalImagesPullingTime="3.125789861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:47:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:47:48.918161004 +0000 UTC m=+291.560124915" watchObservedRunningTime="2026-07-24 16:47:48.918528501 +0000 UTC m=+291.560492385"
Jul 24 16:47:49.319482 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:49.319387    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-ddklx"
Jul 24 16:47:49.493179 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:49.493145    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:49.604068 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:49.603981    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:49.806790 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:49.806731    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:50.168595 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:50.168564    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:50.268674 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:50.268645    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:50.524567 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:50.524527    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cdmqk"]
Jul 24 16:47:50.528103 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:50.528086    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cdmqk"
Jul 24 16:47:50.531671 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:50.531490    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-mqn76\""
Jul 24 16:47:50.531671 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:50.531492    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 24 16:47:50.544916 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:50.544889    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cdmqk"]
Jul 24 16:47:50.548982 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:50.548833    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9e6dcdc5-d8ee-47ca-ad4a-e9262d634f1c-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-cdmqk\" (UID: \"9e6dcdc5-d8ee-47ca-ad4a-e9262d634f1c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cdmqk"
Jul 24 16:47:50.622583 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:50.622549    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:50.649235 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:50.649204    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9e6dcdc5-d8ee-47ca-ad4a-e9262d634f1c-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-cdmqk\" (UID: \"9e6dcdc5-d8ee-47ca-ad4a-e9262d634f1c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cdmqk"
Jul 24 16:47:50.651903 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:50.651879    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9e6dcdc5-d8ee-47ca-ad4a-e9262d634f1c-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-cdmqk\" (UID: \"9e6dcdc5-d8ee-47ca-ad4a-e9262d634f1c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cdmqk"
Jul 24 16:47:50.715533 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:50.715488    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:50.832202 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:50.832117    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:50.840168 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:50.840139    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cdmqk"
Jul 24 16:47:50.985841 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:50.985813    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cdmqk"]
Jul 24 16:47:50.990372 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:47:50.990337    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e6dcdc5_d8ee_47ca_ad4a_e9262d634f1c.slice/crio-5186b9b369d59446463607d467a0589f673c22b907291551f9a9b7c1a600d678 WatchSource:0}: Error finding container 5186b9b369d59446463607d467a0589f673c22b907291551f9a9b7c1a600d678: Status 404 returned error can't find the container with id 5186b9b369d59446463607d467a0589f673c22b907291551f9a9b7c1a600d678
Jul 24 16:47:51.001612 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:51.001580    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-8lwz8" podUID="54f4b6d0-be12-40ae-b3ac-f7acb8e715b9"
Jul 24 16:47:51.018480 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:51.018454    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:51.119060 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:51.118975    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:51.376897 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:51.376768    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:51.597799 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:51.597727    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:51.917031 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:51.916931    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j7rx8" event={"ID":"3c99ab05-563d-4bc8-882a-49defb5b97a9","Type":"ContainerStarted","Data":"d92a9176dea53400f7efcba7c09d1ab8339a96aa3cd7ccf40d616dd744901ef1"}
Jul 24 16:47:51.918357 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:51.918286    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cdmqk" event={"ID":"9e6dcdc5-d8ee-47ca-ad4a-e9262d634f1c","Type":"ContainerStarted","Data":"5186b9b369d59446463607d467a0589f673c22b907291551f9a9b7c1a600d678"}
Jul 24 16:47:51.919524 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:51.919475    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:52.798612 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:52.798580    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:52.899154 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:52.899120    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:52.923628 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:52.923587    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cdmqk" event={"ID":"9e6dcdc5-d8ee-47ca-ad4a-e9262d634f1c","Type":"ContainerStarted","Data":"adf5ad099f5cc5492e15a0855b16a125c14abdf6652b3ea1fab7230629323330"}
Jul 24 16:47:52.923869 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:52.923843    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cdmqk"
Jul 24 16:47:52.928998 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:52.928979    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cdmqk"
Jul 24 16:47:52.938900 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:52.938850    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-cdmqk" podStartSLOduration=1.562157915 podStartE2EDuration="2.938831008s" totalImagesPullingTime="1.376673093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:47:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:47:52.938084675 +0000 UTC m=+295.580048579" watchObservedRunningTime="2026-07-24 16:47:52.938831008 +0000 UTC m=+295.580794895"
Jul 24 16:47:53.634820 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:53.634780    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:53.669128 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:53.669097    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-k5tn8"]
Jul 24 16:47:53.672554 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:53.672535    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-k5tn8"
Jul 24 16:47:53.674966 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:53.674930    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 24 16:47:53.675097 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:53.674927    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 24 16:47:53.675097 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:53.674985    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 24 16:47:53.675826 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:53.675807    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 24 16:47:53.675826 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:53.675821    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-5cnjp\""
Jul 24 16:47:53.675977 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:53.675850    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 24 16:47:53.683236 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:53.681755    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-k5tn8"]
Jul 24 16:47:53.699780 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:53.699760    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:53.779827 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:53.779796    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c796c75a-b5a5-4157-ae11-7474151d0327-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-k5tn8\" (UID: \"c796c75a-b5a5-4157-ae11-7474151d0327\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-k5tn8"
Jul 24 16:47:53.779827 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:53.779829    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsfc5\" (UniqueName: \"kubernetes.io/projected/c796c75a-b5a5-4157-ae11-7474151d0327-kube-api-access-qsfc5\") pod \"prometheus-operator-84d69f794d-k5tn8\" (UID: \"c796c75a-b5a5-4157-ae11-7474151d0327\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-k5tn8"
Jul 24 16:47:53.780025 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:53.779882    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c796c75a-b5a5-4157-ae11-7474151d0327-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-k5tn8\" (UID: \"c796c75a-b5a5-4157-ae11-7474151d0327\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-k5tn8"
Jul 24 16:47:53.780025 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:53.779976    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c796c75a-b5a5-4157-ae11-7474151d0327-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-k5tn8\" (UID: \"c796c75a-b5a5-4157-ae11-7474151d0327\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-k5tn8"
Jul 24 16:47:53.880751 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:53.880690    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c796c75a-b5a5-4157-ae11-7474151d0327-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-k5tn8\" (UID: \"c796c75a-b5a5-4157-ae11-7474151d0327\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-k5tn8"
Jul 24 16:47:53.881248 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:53.880798    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c796c75a-b5a5-4157-ae11-7474151d0327-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-k5tn8\" (UID: \"c796c75a-b5a5-4157-ae11-7474151d0327\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-k5tn8"
Jul 24 16:47:53.881248 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:53.880867    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c796c75a-b5a5-4157-ae11-7474151d0327-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-k5tn8\" (UID: \"c796c75a-b5a5-4157-ae11-7474151d0327\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-k5tn8"
Jul 24 16:47:53.881248 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:53.880892    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsfc5\" (UniqueName: \"kubernetes.io/projected/c796c75a-b5a5-4157-ae11-7474151d0327-kube-api-access-qsfc5\") pod \"prometheus-operator-84d69f794d-k5tn8\" (UID: \"c796c75a-b5a5-4157-ae11-7474151d0327\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-k5tn8"
Jul 24 16:47:53.881248 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:53.880954    2563 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 24 16:47:53.881248 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:53.881028    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c796c75a-b5a5-4157-ae11-7474151d0327-prometheus-operator-tls podName:c796c75a-b5a5-4157-ae11-7474151d0327 nodeName:}" failed. No retries permitted until 2026-07-24 16:47:54.381007428 +0000 UTC m=+297.022971312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/c796c75a-b5a5-4157-ae11-7474151d0327-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-k5tn8" (UID: "c796c75a-b5a5-4157-ae11-7474151d0327") : secret "prometheus-operator-tls" not found
Jul 24 16:47:53.881605 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:53.881588    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c796c75a-b5a5-4157-ae11-7474151d0327-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-k5tn8\" (UID: \"c796c75a-b5a5-4157-ae11-7474151d0327\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-k5tn8"
Jul 24 16:47:53.883504 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:53.883475    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c796c75a-b5a5-4157-ae11-7474151d0327-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-k5tn8\" (UID: \"c796c75a-b5a5-4157-ae11-7474151d0327\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-k5tn8"
Jul 24 16:47:53.889165 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:53.889089    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsfc5\" (UniqueName: \"kubernetes.io/projected/c796c75a-b5a5-4157-ae11-7474151d0327-kube-api-access-qsfc5\") pod \"prometheus-operator-84d69f794d-k5tn8\" (UID: \"c796c75a-b5a5-4157-ae11-7474151d0327\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-k5tn8"
Jul 24 16:47:53.989835 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:53.989802    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:54.385340 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:54.385311    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c796c75a-b5a5-4157-ae11-7474151d0327-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-k5tn8\" (UID: \"c796c75a-b5a5-4157-ae11-7474151d0327\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-k5tn8"
Jul 24 16:47:54.387747 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:54.387715    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c796c75a-b5a5-4157-ae11-7474151d0327-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-k5tn8\" (UID: \"c796c75a-b5a5-4157-ae11-7474151d0327\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-k5tn8"
Jul 24 16:47:54.584276 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:54.584243    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-k5tn8"
Jul 24 16:47:54.708170 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:54.708147    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-k5tn8"]
Jul 24 16:47:54.710307 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:47:54.710286    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc796c75a_b5a5_4157_ae11_7474151d0327.slice/crio-cb6fe355330de5cd675336f40481963176a7e5a9b198214882bee3f078474d53 WatchSource:0}: Error finding container cb6fe355330de5cd675336f40481963176a7e5a9b198214882bee3f078474d53: Status 404 returned error can't find the container with id cb6fe355330de5cd675336f40481963176a7e5a9b198214882bee3f078474d53
Jul 24 16:47:54.932580 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:54.932495    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j7rx8" event={"ID":"3c99ab05-563d-4bc8-882a-49defb5b97a9","Type":"ContainerStarted","Data":"4840d9d56f88e62b171db8cb662bb96d9dc34a41018dfcd2008b45158d6fb4d8"}
Jul 24 16:47:54.933915 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:54.933883    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-k5tn8" event={"ID":"c796c75a-b5a5-4157-ae11-7474151d0327","Type":"ContainerStarted","Data":"cb6fe355330de5cd675336f40481963176a7e5a9b198214882bee3f078474d53"}
Jul 24 16:47:55.895385 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:55.895349    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vrscm"
Jul 24 16:47:56.370225 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:56.370191    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:56.945228 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:56.945190    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-k5tn8" event={"ID":"c796c75a-b5a5-4157-ae11-7474151d0327","Type":"ContainerStarted","Data":"1b47d2cfaa2329f731afa1d7a2f266c42abba31843675f9e2d93b1d67e404a71"}
Jul 24 16:47:56.945418 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:56.945261    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-k5tn8" event={"ID":"c796c75a-b5a5-4157-ae11-7474151d0327","Type":"ContainerStarted","Data":"cdd6debbfc44f306f7d3a1ccc060f2799da66f362ef03382d62ab37470a50594"}
Jul 24 16:47:56.961309 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:56.961263    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-k5tn8" podStartSLOduration=2.616134909 podStartE2EDuration="3.9612448s" totalImagesPullingTime="1.345109891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:47:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:47:56.960192732 +0000 UTC m=+299.602156627" watchObservedRunningTime="2026-07-24 16:47:56.9612448 +0000 UTC m=+299.603208685"
Jul 24 16:47:57.929896 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:57.929868    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ddklx_fb5a9838-1e05-4f86-b46c-760bd7aaa038/console-operator/1.log"
Jul 24 16:47:57.930363 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:57.930092    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ddklx_fb5a9838-1e05-4f86-b46c-760bd7aaa038/console-operator/1.log"
Jul 24 16:47:57.939948 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:57.939919    2563 kubelet.go:1648] "Image garbage collection succeeded"
Jul 24 16:47:58.780418 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:58.780385    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:58.955688 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:58.955648    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j7rx8" event={"ID":"3c99ab05-563d-4bc8-882a-49defb5b97a9","Type":"ContainerStarted","Data":"10ae6918767d205ccdef4b1521cb895e12f0c1d570f421e6f1e4154b27ab594b"}
Jul 24 16:47:58.975708 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:58.975659    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-j7rx8" podStartSLOduration=18.159178928 podStartE2EDuration="29.975643748s" totalImagesPullingTime="11.81646482s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 16:47:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:47:58.974999884 +0000 UTC m=+301.616963789" watchObservedRunningTime="2026-07-24 16:47:58.975643748 +0000 UTC m=+301.617607631"
Jul 24 16:47:59.018032 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.018007    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:59.038494 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.036509    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-kw8ln"]
Jul 24 16:47:59.041365 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.041346    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-kw8ln"
Jul 24 16:47:59.043585 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.043564    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8fr8p"]
Jul 24 16:47:59.043780 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.043765    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 24 16:47:59.044049 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.044032    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 24 16:47:59.044126 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.044032    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-74nd5\""
Jul 24 16:47:59.046529 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.046515    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8fr8p"
Jul 24 16:47:59.049102 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.049084    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 24 16:47:59.049195 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.049148    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9scxg\""
Jul 24 16:47:59.049470 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.049452    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 24 16:47:59.049652 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.049638    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 24 16:47:59.050928 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.050909    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-kw8ln"]
Jul 24 16:47:59.059673 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.059654    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-2flbb"]
Jul 24 16:47:59.062642 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.062626    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-2flbb"
Jul 24 16:47:59.065545 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.065523    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-gvxhl\""
Jul 24 16:47:59.065659 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.065523    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 24 16:47:59.065659 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.065585    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 24 16:47:59.065659 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.065593    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 24 16:47:59.073715 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.073692    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-2flbb"]
Jul 24 16:47:59.098186 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.098163    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:59.125500 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.125471    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rhpd\" (UniqueName: \"kubernetes.io/projected/ab6c9a81-da59-4986-9ec6-74c5197d229a-kube-api-access-7rhpd\") pod \"node-exporter-8fr8p\" (UID: \"ab6c9a81-da59-4986-9ec6-74c5197d229a\") " pod="openshift-monitoring/node-exporter-8fr8p"
Jul 24 16:47:59.125585 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.125507    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ab6c9a81-da59-4986-9ec6-74c5197d229a-node-exporter-accelerators-collector-config\") pod \"node-exporter-8fr8p\" (UID: \"ab6c9a81-da59-4986-9ec6-74c5197d229a\") " pod="openshift-monitoring/node-exporter-8fr8p"
Jul 24 16:47:59.125585 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.125529    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ab6c9a81-da59-4986-9ec6-74c5197d229a-node-exporter-wtmp\") pod \"node-exporter-8fr8p\" (UID: \"ab6c9a81-da59-4986-9ec6-74c5197d229a\") " pod="openshift-monitoring/node-exporter-8fr8p"
Jul 24 16:47:59.125585 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.125573    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-2flbb\" (UID: \"6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-2flbb"
Jul 24 16:47:59.125699 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.125600    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c73dc3e-5e79-4af4-8dfe-b8f78849e114-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-kw8ln\" (UID: \"6c73dc3e-5e79-4af4-8dfe-b8f78849e114\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-kw8ln"
Jul 24 16:47:59.125699 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.125636    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ab6c9a81-da59-4986-9ec6-74c5197d229a-node-exporter-tls\") pod \"node-exporter-8fr8p\" (UID: \"ab6c9a81-da59-4986-9ec6-74c5197d229a\") " pod="openshift-monitoring/node-exporter-8fr8p"
Jul 24 16:47:59.125699 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.125656    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab6c9a81-da59-4986-9ec6-74c5197d229a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8fr8p\" (UID: \"ab6c9a81-da59-4986-9ec6-74c5197d229a\") " pod="openshift-monitoring/node-exporter-8fr8p"
Jul 24 16:47:59.125699 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.125686    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab6c9a81-da59-4986-9ec6-74c5197d229a-metrics-client-ca\") pod \"node-exporter-8fr8p\" (UID: \"ab6c9a81-da59-4986-9ec6-74c5197d229a\") " pod="openshift-monitoring/node-exporter-8fr8p"
Jul 24 16:47:59.125834 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.125735    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-2flbb\" (UID: \"6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-2flbb"
Jul 24 16:47:59.125834 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.125753    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab6c9a81-da59-4986-9ec6-74c5197d229a-sys\") pod \"node-exporter-8fr8p\" (UID: \"ab6c9a81-da59-4986-9ec6-74c5197d229a\") " pod="openshift-monitoring/node-exporter-8fr8p"
Jul 24 16:47:59.125834 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.125769    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-2flbb\" (UID: \"6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-2flbb"
Jul 24 16:47:59.125834 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.125807    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-2flbb\" (UID: \"6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-2flbb"
Jul 24 16:47:59.125834 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.125829    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c73dc3e-5e79-4af4-8dfe-b8f78849e114-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-kw8ln\" (UID: \"6c73dc3e-5e79-4af4-8dfe-b8f78849e114\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-kw8ln"
Jul 24 16:47:59.126034 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.125848    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ab6c9a81-da59-4986-9ec6-74c5197d229a-root\") pod \"node-exporter-8fr8p\" (UID: \"ab6c9a81-da59-4986-9ec6-74c5197d229a\") " pod="openshift-monitoring/node-exporter-8fr8p"
Jul 24 16:47:59.126034 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.125864    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ab6c9a81-da59-4986-9ec6-74c5197d229a-node-exporter-textfile\") pod \"node-exporter-8fr8p\" (UID: \"ab6c9a81-da59-4986-9ec6-74c5197d229a\") " pod="openshift-monitoring/node-exporter-8fr8p"
Jul 24 16:47:59.126034 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.125888    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzfrz\" (UniqueName: \"kubernetes.io/projected/6c73dc3e-5e79-4af4-8dfe-b8f78849e114-kube-api-access-xzfrz\") pod \"openshift-state-metrics-b8f49f5c6-kw8ln\" (UID: \"6c73dc3e-5e79-4af4-8dfe-b8f78849e114\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-kw8ln"
Jul 24 16:47:59.126034 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.125924    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6c73dc3e-5e79-4af4-8dfe-b8f78849e114-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-kw8ln\" (UID: \"6c73dc3e-5e79-4af4-8dfe-b8f78849e114\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-kw8ln"
Jul 24 16:47:59.126034 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.125964    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hccgh\" (UniqueName: \"kubernetes.io/projected/6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657-kube-api-access-hccgh\") pod \"kube-state-metrics-56d9f944b4-2flbb\" (UID: \"6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-2flbb"
Jul 24 16:47:59.126034 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.126004    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-2flbb\" (UID: \"6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-2flbb"
Jul 24 16:47:59.167818 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.167795    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:59.226373 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.226349    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ab6c9a81-da59-4986-9ec6-74c5197d229a-node-exporter-tls\") pod \"node-exporter-8fr8p\" (UID: \"ab6c9a81-da59-4986-9ec6-74c5197d229a\") " pod="openshift-monitoring/node-exporter-8fr8p"
Jul 24 16:47:59.226470 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.226382    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab6c9a81-da59-4986-9ec6-74c5197d229a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8fr8p\" (UID: \"ab6c9a81-da59-4986-9ec6-74c5197d229a\") " pod="openshift-monitoring/node-exporter-8fr8p"
Jul 24 16:47:59.226470 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.226412    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab6c9a81-da59-4986-9ec6-74c5197d229a-metrics-client-ca\") pod \"node-exporter-8fr8p\" (UID: \"ab6c9a81-da59-4986-9ec6-74c5197d229a\") " pod="openshift-monitoring/node-exporter-8fr8p"
Jul 24 16:47:59.226470 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.226438    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-2flbb\" (UID: \"6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-2flbb"
Jul 24 16:47:59.226568 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:59.226503    2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 24 16:47:59.226604 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:47:59.226576    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab6c9a81-da59-4986-9ec6-74c5197d229a-node-exporter-tls podName:ab6c9a81-da59-4986-9ec6-74c5197d229a nodeName:}" failed. No retries permitted until 2026-07-24 16:47:59.72655766 +0000 UTC m=+302.368521527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/ab6c9a81-da59-4986-9ec6-74c5197d229a-node-exporter-tls") pod "node-exporter-8fr8p" (UID: "ab6c9a81-da59-4986-9ec6-74c5197d229a") : secret "node-exporter-tls" not found
Jul 24 16:47:59.226650 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.226600    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab6c9a81-da59-4986-9ec6-74c5197d229a-sys\") pod \"node-exporter-8fr8p\" (UID: \"ab6c9a81-da59-4986-9ec6-74c5197d229a\") " pod="openshift-monitoring/node-exporter-8fr8p"
Jul 24 16:47:59.226650 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.226636    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-2flbb\" (UID: \"6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-2flbb"
Jul 24 16:47:59.226744 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.226672    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-2flbb\" (UID: \"6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-2flbb"
Jul 24 16:47:59.226744 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.226698    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c73dc3e-5e79-4af4-8dfe-b8f78849e114-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-kw8ln\" (UID: \"6c73dc3e-5e79-4af4-8dfe-b8f78849e114\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-kw8ln"
Jul 24 16:47:59.226744 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.226707    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab6c9a81-da59-4986-9ec6-74c5197d229a-sys\") pod \"node-exporter-8fr8p\" (UID: \"ab6c9a81-da59-4986-9ec6-74c5197d229a\") " pod="openshift-monitoring/node-exporter-8fr8p"
Jul 24 16:47:59.226744 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.226722    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ab6c9a81-da59-4986-9ec6-74c5197d229a-root\") pod \"node-exporter-8fr8p\" (UID: \"ab6c9a81-da59-4986-9ec6-74c5197d229a\") " pod="openshift-monitoring/node-exporter-8fr8p"
Jul 24 16:47:59.226932 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.226762    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ab6c9a81-da59-4986-9ec6-74c5197d229a-node-exporter-textfile\") pod \"node-exporter-8fr8p\" (UID: \"ab6c9a81-da59-4986-9ec6-74c5197d229a\") " pod="openshift-monitoring/node-exporter-8fr8p"
Jul 24 16:47:59.226932 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.226769    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ab6c9a81-da59-4986-9ec6-74c5197d229a-root\") pod \"node-exporter-8fr8p\" (UID: \"ab6c9a81-da59-4986-9ec6-74c5197d229a\") " pod="openshift-monitoring/node-exporter-8fr8p"
Jul 24 16:47:59.226932 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.226785    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-2flbb\" (UID: \"6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-2flbb"
Jul 24 16:47:59.226932 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.226801    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzfrz\" (UniqueName: \"kubernetes.io/projected/6c73dc3e-5e79-4af4-8dfe-b8f78849e114-kube-api-access-xzfrz\") pod \"openshift-state-metrics-b8f49f5c6-kw8ln\" (UID: \"6c73dc3e-5e79-4af4-8dfe-b8f78849e114\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-kw8ln"
Jul 24 16:47:59.226932 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.226833    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6c73dc3e-5e79-4af4-8dfe-b8f78849e114-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-kw8ln\" (UID: \"6c73dc3e-5e79-4af4-8dfe-b8f78849e114\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-kw8ln"
Jul 24 16:47:59.226932 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.226858    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hccgh\" (UniqueName: \"kubernetes.io/projected/6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657-kube-api-access-hccgh\") pod \"kube-state-metrics-56d9f944b4-2flbb\" (UID: \"6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-2flbb"
Jul 24 16:47:59.226932 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.226888    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-2flbb\" (UID: \"6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-2flbb"
Jul 24 16:47:59.227320 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.226963    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rhpd\" (UniqueName: \"kubernetes.io/projected/ab6c9a81-da59-4986-9ec6-74c5197d229a-kube-api-access-7rhpd\") pod \"node-exporter-8fr8p\" (UID: \"ab6c9a81-da59-4986-9ec6-74c5197d229a\") " pod="openshift-monitoring/node-exporter-8fr8p"
Jul 24 16:47:59.227320 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.227003    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ab6c9a81-da59-4986-9ec6-74c5197d229a-node-exporter-accelerators-collector-config\") pod \"node-exporter-8fr8p\" (UID: \"ab6c9a81-da59-4986-9ec6-74c5197d229a\") " pod="openshift-monitoring/node-exporter-8fr8p"
Jul 24 16:47:59.227320 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.227036    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ab6c9a81-da59-4986-9ec6-74c5197d229a-node-exporter-wtmp\") pod \"node-exporter-8fr8p\" (UID: \"ab6c9a81-da59-4986-9ec6-74c5197d229a\") " pod="openshift-monitoring/node-exporter-8fr8p"
Jul 24 16:47:59.227320 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.227061    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-2flbb\" (UID: \"6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-2flbb"
Jul 24 16:47:59.227320 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.227085    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ab6c9a81-da59-4986-9ec6-74c5197d229a-node-exporter-textfile\") pod \"node-exporter-8fr8p\" (UID: \"ab6c9a81-da59-4986-9ec6-74c5197d229a\") " pod="openshift-monitoring/node-exporter-8fr8p"
Jul 24 16:47:59.227320 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.227089    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c73dc3e-5e79-4af4-8dfe-b8f78849e114-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-kw8ln\" (UID: \"6c73dc3e-5e79-4af4-8dfe-b8f78849e114\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-kw8ln"
Jul 24 16:47:59.227320 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.227232    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab6c9a81-da59-4986-9ec6-74c5197d229a-metrics-client-ca\") pod \"node-exporter-8fr8p\" (UID: \"ab6c9a81-da59-4986-9ec6-74c5197d229a\") " pod="openshift-monitoring/node-exporter-8fr8p"
Jul 24 16:47:59.227662 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.227523    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c73dc3e-5e79-4af4-8dfe-b8f78849e114-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-kw8ln\" (UID: \"6c73dc3e-5e79-4af4-8dfe-b8f78849e114\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-kw8ln"
Jul 24 16:47:59.227662 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.227568    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-2flbb\" (UID: \"6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-2flbb"
Jul 24 16:47:59.227766 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.227660    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ab6c9a81-da59-4986-9ec6-74c5197d229a-node-exporter-wtmp\") pod \"node-exporter-8fr8p\" (UID: \"ab6c9a81-da59-4986-9ec6-74c5197d229a\") " pod="openshift-monitoring/node-exporter-8fr8p"
Jul 24 16:47:59.227766 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.227693    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ab6c9a81-da59-4986-9ec6-74c5197d229a-node-exporter-accelerators-collector-config\") pod \"node-exporter-8fr8p\" (UID: \"ab6c9a81-da59-4986-9ec6-74c5197d229a\") " pod="openshift-monitoring/node-exporter-8fr8p"
Jul 24 16:47:59.228310 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.228290    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-2flbb\" (UID: \"6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-2flbb"
Jul 24 16:47:59.229246 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.229228    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab6c9a81-da59-4986-9ec6-74c5197d229a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8fr8p\" (UID: \"ab6c9a81-da59-4986-9ec6-74c5197d229a\") " pod="openshift-monitoring/node-exporter-8fr8p"
Jul 24 16:47:59.229722 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.229691    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-2flbb\" (UID: \"6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-2flbb"
Jul 24 16:47:59.229803 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.229729    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-2flbb\" (UID: \"6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-2flbb"
Jul 24 16:47:59.229986 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.229962    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c73dc3e-5e79-4af4-8dfe-b8f78849e114-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-kw8ln\" (UID: \"6c73dc3e-5e79-4af4-8dfe-b8f78849e114\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-kw8ln"
Jul 24 16:47:59.230059 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.230045    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6c73dc3e-5e79-4af4-8dfe-b8f78849e114-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-kw8ln\" (UID: \"6c73dc3e-5e79-4af4-8dfe-b8f78849e114\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-kw8ln"
Jul 24 16:47:59.236278 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.236257    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rhpd\" (UniqueName: \"kubernetes.io/projected/ab6c9a81-da59-4986-9ec6-74c5197d229a-kube-api-access-7rhpd\") pod \"node-exporter-8fr8p\" (UID: \"ab6c9a81-da59-4986-9ec6-74c5197d229a\") " pod="openshift-monitoring/node-exporter-8fr8p"
Jul 24 16:47:59.236375 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.236285    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzfrz\" (UniqueName: \"kubernetes.io/projected/6c73dc3e-5e79-4af4-8dfe-b8f78849e114-kube-api-access-xzfrz\") pod \"openshift-state-metrics-b8f49f5c6-kw8ln\" (UID: \"6c73dc3e-5e79-4af4-8dfe-b8f78849e114\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-kw8ln"
Jul 24 16:47:59.236375 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.236349    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hccgh\" (UniqueName: \"kubernetes.io/projected/6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657-kube-api-access-hccgh\") pod \"kube-state-metrics-56d9f944b4-2flbb\" (UID: \"6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-2flbb"
Jul 24 16:47:59.268792 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.268759    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:59.325143 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.325067    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:59.353098 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.353077    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-kw8ln"
Jul 24 16:47:59.371537 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.371511    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-2flbb"
Jul 24 16:47:59.391985 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.391929    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:59.491051 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.491017    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:59.526825 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.526783    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-kw8ln"]
Jul 24 16:47:59.529251 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:47:59.529227    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c73dc3e_5e79_4af4_8dfe_b8f78849e114.slice/crio-d17f4b49d21ba2f69c898c216e8e3ba6b1142e69268272ae671d513aa5a7e9eb WatchSource:0}: Error finding container d17f4b49d21ba2f69c898c216e8e3ba6b1142e69268272ae671d513aa5a7e9eb: Status 404 returned error can't find the container with id d17f4b49d21ba2f69c898c216e8e3ba6b1142e69268272ae671d513aa5a7e9eb
Jul 24 16:47:59.542313 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.542230    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-2flbb"]
Jul 24 16:47:59.544800 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:47:59.544777    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae0d5ba_9ae1_44ec_ac8f_6c0ae405b657.slice/crio-1c0d8ae7616dce0b31ae81d0e4055a5fec02f06f072b3c2d67531b05a850c359 WatchSource:0}: Error finding container 1c0d8ae7616dce0b31ae81d0e4055a5fec02f06f072b3c2d67531b05a850c359: Status 404 returned error can't find the container with id 1c0d8ae7616dce0b31ae81d0e4055a5fec02f06f072b3c2d67531b05a850c359
Jul 24 16:47:59.546607 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.546593    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 16:47:59.549673 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.549655    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:47:59.732207 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.732176    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ab6c9a81-da59-4986-9ec6-74c5197d229a-node-exporter-tls\") pod \"node-exporter-8fr8p\" (UID: \"ab6c9a81-da59-4986-9ec6-74c5197d229a\") " pod="openshift-monitoring/node-exporter-8fr8p"
Jul 24 16:47:59.734426 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.734400    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ab6c9a81-da59-4986-9ec6-74c5197d229a-node-exporter-tls\") pod \"node-exporter-8fr8p\" (UID: \"ab6c9a81-da59-4986-9ec6-74c5197d229a\") " pod="openshift-monitoring/node-exporter-8fr8p"
Jul 24 16:47:59.957911 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.957883    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8fr8p"
Jul 24 16:47:59.961490 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.961462    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-kw8ln" event={"ID":"6c73dc3e-5e79-4af4-8dfe-b8f78849e114","Type":"ContainerStarted","Data":"2fb945618bc0522ccc76624dfa4ed4ecf7e142f92ab41e872ebf129b2472e3af"}
Jul 24 16:47:59.961594 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.961539    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-kw8ln" event={"ID":"6c73dc3e-5e79-4af4-8dfe-b8f78849e114","Type":"ContainerStarted","Data":"ad136926ac1257894a495c59058866c24d6a1871ed7b509f9df5bcc5128c859f"}
Jul 24 16:47:59.961594 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.961574    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-kw8ln" event={"ID":"6c73dc3e-5e79-4af4-8dfe-b8f78849e114","Type":"ContainerStarted","Data":"d17f4b49d21ba2f69c898c216e8e3ba6b1142e69268272ae671d513aa5a7e9eb"}
Jul 24 16:47:59.963535 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:47:59.963054    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-2flbb" event={"ID":"6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657","Type":"ContainerStarted","Data":"1c0d8ae7616dce0b31ae81d0e4055a5fec02f06f072b3c2d67531b05a850c359"}
Jul 24 16:47:59.968897 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:47:59.968875    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab6c9a81_da59_4986_9ec6_74c5197d229a.slice/crio-f8cc52f581c327575055cdc4d879f567915e392d9ee113e46460433ec7040ddb WatchSource:0}: Error finding container f8cc52f581c327575055cdc4d879f567915e392d9ee113e46460433ec7040ddb: Status 404 returned error can't find the container with id f8cc52f581c327575055cdc4d879f567915e392d9ee113e46460433ec7040ddb
Jul 24 16:48:00.015482 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:00.015441    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:00.968570 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:00.968535    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8fr8p" event={"ID":"ab6c9a81-da59-4986-9ec6-74c5197d229a","Type":"ContainerStarted","Data":"f8cc52f581c327575055cdc4d879f567915e392d9ee113e46460433ec7040ddb"}
Jul 24 16:48:01.227112 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:01.227023    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:01.935849 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:01.935816    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:01.973532 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:01.973500    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-2flbb" event={"ID":"6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657","Type":"ContainerStarted","Data":"edcc5660f7469f00b5b8350b929c7710c2e9dc614145543b397651804cbd4b08"}
Jul 24 16:48:01.973976 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:01.973560    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-2flbb" event={"ID":"6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657","Type":"ContainerStarted","Data":"dda34b7c97a38f9268a35008f65923fd8dcd1b8424566dfd2b6994e046e6df5e"}
Jul 24 16:48:01.973976 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:01.973583    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-2flbb" event={"ID":"6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657","Type":"ContainerStarted","Data":"4abb93c3a0fc8f3d6fbf7c6e03ce56f9650e4e2737df9672d5d94eed18cec8b0"}
Jul 24 16:48:01.975058 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:01.975036    2563 generic.go:356] "Generic (PLEG): container finished" podID="ab6c9a81-da59-4986-9ec6-74c5197d229a" containerID="0671679ea5b61ff0bd1b44e3c0112be9ec5e9dae7c81e3809659b9323c44b556" exitCode=0
Jul 24 16:48:01.975164 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:01.975123    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8fr8p" event={"ID":"ab6c9a81-da59-4986-9ec6-74c5197d229a","Type":"ContainerDied","Data":"0671679ea5b61ff0bd1b44e3c0112be9ec5e9dae7c81e3809659b9323c44b556"}
Jul 24 16:48:01.977319 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:01.977300    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-kw8ln" event={"ID":"6c73dc3e-5e79-4af4-8dfe-b8f78849e114","Type":"ContainerStarted","Data":"cbf8e245c12f1526ffed8d46761efd37706f840a4f99cd967b893d4e8a1e6e1b"}
Jul 24 16:48:01.997685 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:01.994161    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-2flbb" podStartSLOduration=1.236131873 podStartE2EDuration="2.994142447s" totalImagesPullingTime="1.758010574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:47:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:48:01.990921237 +0000 UTC m=+304.632885123" watchObservedRunningTime="2026-07-24 16:48:01.994142447 +0000 UTC m=+304.636106334"
Jul 24 16:48:01.999074 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:01.999038    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:02.023873 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:02.023826    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-kw8ln" podStartSLOduration=1.369211105 podStartE2EDuration="3.023807507s" totalImagesPullingTime="1.654596402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:47:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:48:02.023574594 +0000 UTC m=+304.665538499" watchObservedRunningTime="2026-07-24 16:48:02.023807507 +0000 UTC m=+304.665771391"
Jul 24 16:48:02.466974 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:02.466915    2563 patch_prober.go:28] interesting pod/image-registry-c8f78f7cf-2sdpb container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 24 16:48:02.467143 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:02.466992    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb" podUID="6e7f3a6f-1da1-4a07-b736-512dfb2afd6a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 16:48:02.516445 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:02.516414    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:02.623225 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:02.623189    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:02.984908 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:02.984861    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8fr8p" event={"ID":"ab6c9a81-da59-4986-9ec6-74c5197d229a","Type":"ContainerStarted","Data":"dea3c391b9283f1309482a2a6845a4b4336ef0d1df51cabd68c5f1263ed730ba"}
Jul 24 16:48:02.985358 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:02.984931    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8fr8p" event={"ID":"ab6c9a81-da59-4986-9ec6-74c5197d229a","Type":"ContainerStarted","Data":"cdf7fe1e336bee969bc4f1f0f38dac79273b75c336098c55a6e9e596948d8af8"}
Jul 24 16:48:03.004989 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:03.004924    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8fr8p" podStartSLOduration=2.592462208 podStartE2EDuration="4.00490931s" totalImagesPullingTime="1.336511476s" totalInitContainerRuntime="75.935626ms" isStatefulPod=true podCreationTimestamp="2026-07-24 16:47:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:48:03.003812247 +0000 UTC m=+305.645776152" watchObservedRunningTime="2026-07-24 16:48:03.00490931 +0000 UTC m=+305.646873172"
Jul 24 16:48:03.799128 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:03.799094    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:03.898275 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:03.898240    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:04.032793 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:04.032763    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:04.884201 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:04.884173    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:48:04.996538 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:04.996508    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lwz8"
Jul 24 16:48:05.081461 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:05.081424    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:05.182913 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:05.182799    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:05.249679 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:05.249648    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:06.616843 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:06.616809    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:06.716699 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:06.716667    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:06.818599 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:06.818564    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:06.919821 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:06.919760    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:10.213085 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:10.213054    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84b84cc59-mhbkn"]
Jul 24 16:48:10.217488 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:10.217464    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84b84cc59-mhbkn"
Jul 24 16:48:10.220679 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:10.220654    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 24 16:48:10.220947 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:10.220910    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 24 16:48:10.220947 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:10.220911    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 24 16:48:10.221078 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:10.220911    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 24 16:48:10.221540 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:10.221519    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 24 16:48:10.221667 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:10.221523    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 24 16:48:10.221667 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:10.221595    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 24 16:48:10.221775 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:10.221719    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-q6qvj\""
Jul 24 16:48:10.233298 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:10.233271    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 24 16:48:10.234092 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:10.234072    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84b84cc59-mhbkn"]
Jul 24 16:48:10.327921 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:10.327887    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd24175c-399c-4773-b526-5e5b63a5e019-trusted-ca-bundle\") pod \"console-84b84cc59-mhbkn\" (UID: \"dd24175c-399c-4773-b526-5e5b63a5e019\") " pod="openshift-console/console-84b84cc59-mhbkn"
Jul 24 16:48:10.327921 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:10.327927    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd24175c-399c-4773-b526-5e5b63a5e019-console-serving-cert\") pod \"console-84b84cc59-mhbkn\" (UID: \"dd24175c-399c-4773-b526-5e5b63a5e019\") " pod="openshift-console/console-84b84cc59-mhbkn"
Jul 24 16:48:10.328125 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:10.327997    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd24175c-399c-4773-b526-5e5b63a5e019-oauth-serving-cert\") pod \"console-84b84cc59-mhbkn\" (UID: \"dd24175c-399c-4773-b526-5e5b63a5e019\") " pod="openshift-console/console-84b84cc59-mhbkn"
Jul 24 16:48:10.328125 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:10.328045    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd24175c-399c-4773-b526-5e5b63a5e019-console-oauth-config\") pod \"console-84b84cc59-mhbkn\" (UID: \"dd24175c-399c-4773-b526-5e5b63a5e019\") " pod="openshift-console/console-84b84cc59-mhbkn"
Jul 24 16:48:10.328125 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:10.328063    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd24175c-399c-4773-b526-5e5b63a5e019-service-ca\") pod \"console-84b84cc59-mhbkn\" (UID: \"dd24175c-399c-4773-b526-5e5b63a5e019\") " pod="openshift-console/console-84b84cc59-mhbkn"
Jul 24 16:48:10.328125 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:10.328111    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd24175c-399c-4773-b526-5e5b63a5e019-console-config\") pod \"console-84b84cc59-mhbkn\" (UID: \"dd24175c-399c-4773-b526-5e5b63a5e019\") " pod="openshift-console/console-84b84cc59-mhbkn"
Jul 24 16:48:10.328242 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:10.328140    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhjwz\" (UniqueName: \"kubernetes.io/projected/dd24175c-399c-4773-b526-5e5b63a5e019-kube-api-access-xhjwz\") pod \"console-84b84cc59-mhbkn\" (UID: \"dd24175c-399c-4773-b526-5e5b63a5e019\") " pod="openshift-console/console-84b84cc59-mhbkn"
Jul 24 16:48:10.429125 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:10.429065    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhjwz\" (UniqueName: \"kubernetes.io/projected/dd24175c-399c-4773-b526-5e5b63a5e019-kube-api-access-xhjwz\") pod \"console-84b84cc59-mhbkn\" (UID: \"dd24175c-399c-4773-b526-5e5b63a5e019\") " pod="openshift-console/console-84b84cc59-mhbkn"
Jul 24 16:48:10.429301 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:10.429165    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd24175c-399c-4773-b526-5e5b63a5e019-trusted-ca-bundle\") pod \"console-84b84cc59-mhbkn\" (UID: \"dd24175c-399c-4773-b526-5e5b63a5e019\") " pod="openshift-console/console-84b84cc59-mhbkn"
Jul 24 16:48:10.429301 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:10.429199    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd24175c-399c-4773-b526-5e5b63a5e019-console-serving-cert\") pod \"console-84b84cc59-mhbkn\" (UID: \"dd24175c-399c-4773-b526-5e5b63a5e019\") " pod="openshift-console/console-84b84cc59-mhbkn"
Jul 24 16:48:10.429301 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:10.429238    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd24175c-399c-4773-b526-5e5b63a5e019-oauth-serving-cert\") pod \"console-84b84cc59-mhbkn\" (UID: \"dd24175c-399c-4773-b526-5e5b63a5e019\") " pod="openshift-console/console-84b84cc59-mhbkn"
Jul 24 16:48:10.429301 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:10.429282    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd24175c-399c-4773-b526-5e5b63a5e019-console-oauth-config\") pod \"console-84b84cc59-mhbkn\" (UID: \"dd24175c-399c-4773-b526-5e5b63a5e019\") " pod="openshift-console/console-84b84cc59-mhbkn"
Jul 24 16:48:10.429514 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:10.429302    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd24175c-399c-4773-b526-5e5b63a5e019-service-ca\") pod \"console-84b84cc59-mhbkn\" (UID: \"dd24175c-399c-4773-b526-5e5b63a5e019\") " pod="openshift-console/console-84b84cc59-mhbkn"
Jul 24 16:48:10.429514 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:10.429352    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd24175c-399c-4773-b526-5e5b63a5e019-console-config\") pod \"console-84b84cc59-mhbkn\" (UID: \"dd24175c-399c-4773-b526-5e5b63a5e019\") " pod="openshift-console/console-84b84cc59-mhbkn"
Jul 24 16:48:10.430142 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:10.430115    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd24175c-399c-4773-b526-5e5b63a5e019-trusted-ca-bundle\") pod \"console-84b84cc59-mhbkn\" (UID: \"dd24175c-399c-4773-b526-5e5b63a5e019\") " pod="openshift-console/console-84b84cc59-mhbkn"
Jul 24 16:48:10.430261 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:10.430119    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd24175c-399c-4773-b526-5e5b63a5e019-console-config\") pod \"console-84b84cc59-mhbkn\" (UID: \"dd24175c-399c-4773-b526-5e5b63a5e019\") " pod="openshift-console/console-84b84cc59-mhbkn"
Jul 24 16:48:10.430261 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:10.430164    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd24175c-399c-4773-b526-5e5b63a5e019-oauth-serving-cert\") pod \"console-84b84cc59-mhbkn\" (UID: \"dd24175c-399c-4773-b526-5e5b63a5e019\") " pod="openshift-console/console-84b84cc59-mhbkn"
Jul 24 16:48:10.430261 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:10.430206    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd24175c-399c-4773-b526-5e5b63a5e019-service-ca\") pod \"console-84b84cc59-mhbkn\" (UID: \"dd24175c-399c-4773-b526-5e5b63a5e019\") " pod="openshift-console/console-84b84cc59-mhbkn"
Jul 24 16:48:10.431795 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:10.431767    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd24175c-399c-4773-b526-5e5b63a5e019-console-serving-cert\") pod \"console-84b84cc59-mhbkn\" (UID: \"dd24175c-399c-4773-b526-5e5b63a5e019\") " pod="openshift-console/console-84b84cc59-mhbkn"
Jul 24 16:48:10.431795 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:10.431779    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd24175c-399c-4773-b526-5e5b63a5e019-console-oauth-config\") pod \"console-84b84cc59-mhbkn\" (UID: \"dd24175c-399c-4773-b526-5e5b63a5e019\") " pod="openshift-console/console-84b84cc59-mhbkn"
Jul 24 16:48:10.437334 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:10.437310    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhjwz\" (UniqueName: \"kubernetes.io/projected/dd24175c-399c-4773-b526-5e5b63a5e019-kube-api-access-xhjwz\") pod \"console-84b84cc59-mhbkn\" (UID: \"dd24175c-399c-4773-b526-5e5b63a5e019\") " pod="openshift-console/console-84b84cc59-mhbkn"
Jul 24 16:48:10.529442 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:10.529376    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84b84cc59-mhbkn"
Jul 24 16:48:10.629567 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:10.629534    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:10.662176 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:10.662087    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84b84cc59-mhbkn"]
Jul 24 16:48:10.664724 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:48:10.664697    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd24175c_399c_4773_b526_5e5b63a5e019.slice/crio-0fef68f74fa428f4541f6c3a8632a0cd90d05697d9ea6d9dd5b2d6019ea0f3ce WatchSource:0}: Error finding container 0fef68f74fa428f4541f6c3a8632a0cd90d05697d9ea6d9dd5b2d6019ea0f3ce: Status 404 returned error can't find the container with id 0fef68f74fa428f4541f6c3a8632a0cd90d05697d9ea6d9dd5b2d6019ea0f3ce
Jul 24 16:48:10.735586 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:10.735553    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:11.015733 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:11.015698    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84b84cc59-mhbkn" event={"ID":"dd24175c-399c-4773-b526-5e5b63a5e019","Type":"ContainerStarted","Data":"0fef68f74fa428f4541f6c3a8632a0cd90d05697d9ea6d9dd5b2d6019ea0f3ce"}
Jul 24 16:48:11.018101 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:11.018072    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:11.118637 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:11.118602    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:12.118451 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:12.118417    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:12.221227 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:12.221179    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:13.518124 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:13.518100    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-c8f78f7cf-2sdpb"]
Jul 24 16:48:13.601510 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:13.601471    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:13.703037 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:13.703006    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:14.030203 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:14.030170    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84b84cc59-mhbkn" event={"ID":"dd24175c-399c-4773-b526-5e5b63a5e019","Type":"ContainerStarted","Data":"a9dee69c3fe1d6af3ba91d1366c65a5c9fe4a73038667f4a4a7fa01a55082001"}
Jul 24 16:48:14.292225 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:14.292144    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:14.393900 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:14.393867    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:15.788577 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:15.788535    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:15.889056 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:15.889025    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:20.010089 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:20.010054    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:20.111031 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:20.110994    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:20.530561 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:20.530531    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84b84cc59-mhbkn"
Jul 24 16:48:20.530742 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:20.530583    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84b84cc59-mhbkn"
Jul 24 16:48:20.535613 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:20.535583    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84b84cc59-mhbkn"
Jul 24 16:48:20.553482 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:20.553440    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84b84cc59-mhbkn" podStartSLOduration=7.746801662 podStartE2EDuration="10.553426686s" totalImagesPullingTime="2.806625024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:48:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:48:14.046172465 +0000 UTC m=+316.688136358" watchObservedRunningTime="2026-07-24 16:48:20.553426686 +0000 UTC m=+323.195390569"
Jul 24 16:48:20.811635 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:20.811558    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:20.913054 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:20.913023    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:21.061264 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:21.061235    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84b84cc59-mhbkn"
Jul 24 16:48:21.200075 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:21.200027    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:21.300991 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:21.300954    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:22.208265 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:22.208226    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:22.261976 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:22.261920    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:23.036034 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:23.036000    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:23.097546 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:23.097508    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:25.167264 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:25.167218    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:25.267853 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:25.267822    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:38.540620 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:38.540557    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb" podUID="6e7f3a6f-1da1-4a07-b736-512dfb2afd6a" containerName="registry" containerID="cri-o://17a2c0edffad5621dae52f44d4069a52dc0d5881865027c4120a92e1dfbfb2ae" gracePeriod=30
Jul 24 16:48:38.786728 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:38.786704    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:48:38.890335 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:38.890302    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-registry-tls\") pod \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") "
Jul 24 16:48:38.890537 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:38.890347    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-bound-sa-token\") pod \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") "
Jul 24 16:48:38.890537 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:38.890376    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-registry-certificates\") pod \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") "
Jul 24 16:48:38.890537 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:38.890421    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-ca-trust-extracted\") pod \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") "
Jul 24 16:48:38.890537 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:38.890447    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-trusted-ca\") pod \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") "
Jul 24 16:48:38.890537 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:38.890496    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-image-registry-private-configuration\") pod \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") "
Jul 24 16:48:38.890537 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:38.890526    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-installation-pull-secrets\") pod \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") "
Jul 24 16:48:38.890845 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:38.890558    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p29fm\" (UniqueName: \"kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-kube-api-access-p29fm\") pod \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\" (UID: \"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a\") "
Jul 24 16:48:38.890926 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:38.890898    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "6e7f3a6f-1da1-4a07-b736-512dfb2afd6a" (UID: "6e7f3a6f-1da1-4a07-b736-512dfb2afd6a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 16:48:38.891014 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:38.890931    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "6e7f3a6f-1da1-4a07-b736-512dfb2afd6a" (UID: "6e7f3a6f-1da1-4a07-b736-512dfb2afd6a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 16:48:38.893296 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:38.893262    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "6e7f3a6f-1da1-4a07-b736-512dfb2afd6a" (UID: "6e7f3a6f-1da1-4a07-b736-512dfb2afd6a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 16:48:38.893296 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:38.893279    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "6e7f3a6f-1da1-4a07-b736-512dfb2afd6a" (UID: "6e7f3a6f-1da1-4a07-b736-512dfb2afd6a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 16:48:38.893581 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:38.893551    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6e7f3a6f-1da1-4a07-b736-512dfb2afd6a" (UID: "6e7f3a6f-1da1-4a07-b736-512dfb2afd6a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 16:48:38.893671 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:38.893551    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "6e7f3a6f-1da1-4a07-b736-512dfb2afd6a" (UID: "6e7f3a6f-1da1-4a07-b736-512dfb2afd6a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 16:48:38.893671 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:38.893645    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-kube-api-access-p29fm" (OuterVolumeSpecName: "kube-api-access-p29fm") pod "6e7f3a6f-1da1-4a07-b736-512dfb2afd6a" (UID: "6e7f3a6f-1da1-4a07-b736-512dfb2afd6a"). InnerVolumeSpecName "kube-api-access-p29fm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 16:48:38.898845 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:38.898820    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "6e7f3a6f-1da1-4a07-b736-512dfb2afd6a" (UID: "6e7f3a6f-1da1-4a07-b736-512dfb2afd6a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:48:38.991871 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:38.991833    2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-ca-trust-extracted\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 16:48:38.991871 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:38.991865    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-trusted-ca\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 16:48:38.991871 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:38.991876    2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-image-registry-private-configuration\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 16:48:38.991871 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:38.991885    2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-installation-pull-secrets\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 16:48:38.992162 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:38.991895    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p29fm\" (UniqueName: \"kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-kube-api-access-p29fm\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 16:48:38.992162 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:38.991903    2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-registry-tls\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 16:48:38.992162 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:38.991911    2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-bound-sa-token\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 16:48:38.992162 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:38.991918    2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a-registry-certificates\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 16:48:39.125038 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:39.125000    2563 generic.go:356] "Generic (PLEG): container finished" podID="6e7f3a6f-1da1-4a07-b736-512dfb2afd6a" containerID="17a2c0edffad5621dae52f44d4069a52dc0d5881865027c4120a92e1dfbfb2ae" exitCode=0
Jul 24 16:48:39.125197 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:39.125052    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb"
Jul 24 16:48:39.125197 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:39.125067    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb" event={"ID":"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a","Type":"ContainerDied","Data":"17a2c0edffad5621dae52f44d4069a52dc0d5881865027c4120a92e1dfbfb2ae"}
Jul 24 16:48:39.125197 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:39.125115    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c8f78f7cf-2sdpb" event={"ID":"6e7f3a6f-1da1-4a07-b736-512dfb2afd6a","Type":"ContainerDied","Data":"e63dd6e8320908a68c4598e679b1854b479a77c1f98840c8f4c0b897f53d992b"}
Jul 24 16:48:39.125197 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:39.125142    2563 scope.go:117] "RemoveContainer" containerID="17a2c0edffad5621dae52f44d4069a52dc0d5881865027c4120a92e1dfbfb2ae"
Jul 24 16:48:39.133884 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:39.133867    2563 scope.go:117] "RemoveContainer" containerID="17a2c0edffad5621dae52f44d4069a52dc0d5881865027c4120a92e1dfbfb2ae"
Jul 24 16:48:39.134156 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:48:39.134134    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17a2c0edffad5621dae52f44d4069a52dc0d5881865027c4120a92e1dfbfb2ae\": container with ID starting with 17a2c0edffad5621dae52f44d4069a52dc0d5881865027c4120a92e1dfbfb2ae not found: ID does not exist" containerID="17a2c0edffad5621dae52f44d4069a52dc0d5881865027c4120a92e1dfbfb2ae"
Jul 24 16:48:39.134223 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:39.134162    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17a2c0edffad5621dae52f44d4069a52dc0d5881865027c4120a92e1dfbfb2ae"} err="failed to get container status \"17a2c0edffad5621dae52f44d4069a52dc0d5881865027c4120a92e1dfbfb2ae\": rpc error: code = NotFound desc = could not find container \"17a2c0edffad5621dae52f44d4069a52dc0d5881865027c4120a92e1dfbfb2ae\": container with ID starting with 17a2c0edffad5621dae52f44d4069a52dc0d5881865027c4120a92e1dfbfb2ae not found: ID does not exist"
Jul 24 16:48:39.145571 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:39.145515    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-c8f78f7cf-2sdpb"]
Jul 24 16:48:39.149214 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:39.149194    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-c8f78f7cf-2sdpb"]
Jul 24 16:48:40.003204 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:40.002653    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e7f3a6f-1da1-4a07-b736-512dfb2afd6a" path="/var/lib/kubelet/pods/6e7f3a6f-1da1-4a07-b736-512dfb2afd6a/volumes"
Jul 24 16:48:44.291426 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:44.291391    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:48:44.391977 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:48:44.391908    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:49:00.202056 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:00.202021    2563 generic.go:356] "Generic (PLEG): container finished" podID="ef1872b5-49d6-4162-bcba-afea4c9ca621" containerID="6c491c3a8390c04e93470b442628b4f7f32ae67572266bd17c06b7087294192d" exitCode=0
Jul 24 16:49:00.202571 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:00.202098    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-5qj6l" event={"ID":"ef1872b5-49d6-4162-bcba-afea4c9ca621","Type":"ContainerDied","Data":"6c491c3a8390c04e93470b442628b4f7f32ae67572266bd17c06b7087294192d"}
Jul 24 16:49:00.202571 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:00.202477    2563 scope.go:117] "RemoveContainer" containerID="6c491c3a8390c04e93470b442628b4f7f32ae67572266bd17c06b7087294192d"
Jul 24 16:49:00.203763 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:00.203671    2563 generic.go:356] "Generic (PLEG): container finished" podID="e3460f62-6389-4673-9e33-e4e5aa0d4d9a" containerID="ea28021250bd740bbb86badbbce6cd94e71973c10cd33ee162f9ff9c52b61f70" exitCode=0
Jul 24 16:49:00.203763 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:00.203744    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xl9vk" event={"ID":"e3460f62-6389-4673-9e33-e4e5aa0d4d9a","Type":"ContainerDied","Data":"ea28021250bd740bbb86badbbce6cd94e71973c10cd33ee162f9ff9c52b61f70"}
Jul 24 16:49:00.204037 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:00.204014    2563 scope.go:117] "RemoveContainer" containerID="ea28021250bd740bbb86badbbce6cd94e71973c10cd33ee162f9ff9c52b61f70"
Jul 24 16:49:01.209627 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:01.209590    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-5qj6l" event={"ID":"ef1872b5-49d6-4162-bcba-afea4c9ca621","Type":"ContainerStarted","Data":"11bcbc336bc32c3f729f379da696c8926c413e7015e710d7da9248c0bc550b62"}
Jul 24 16:49:01.211387 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:01.211361    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xl9vk" event={"ID":"e3460f62-6389-4673-9e33-e4e5aa0d4d9a","Type":"ContainerStarted","Data":"464e3372a56a9805eb5f8d2256a3683f04b24cb6bbe67714b2ca28e7b17081bb"}
Jul 24 16:49:06.669676 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:06.669611    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:49:10.749116 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:10.749078    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54f4b6d0-be12-40ae-b3ac-f7acb8e715b9-metrics-certs\") pod \"network-metrics-daemon-8lwz8\" (UID: \"54f4b6d0-be12-40ae-b3ac-f7acb8e715b9\") " pod="openshift-multus/network-metrics-daemon-8lwz8"
Jul 24 16:49:10.751454 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:10.751430    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54f4b6d0-be12-40ae-b3ac-f7acb8e715b9-metrics-certs\") pod \"network-metrics-daemon-8lwz8\" (UID: \"54f4b6d0-be12-40ae-b3ac-f7acb8e715b9\") " pod="openshift-multus/network-metrics-daemon-8lwz8"
Jul 24 16:49:11.000374 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:11.000294    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6gtf2\""
Jul 24 16:49:11.008233 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:11.008214    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8lwz8"
Jul 24 16:49:11.135011 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:11.134991    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8lwz8"]
Jul 24 16:49:11.138043 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:49:11.138012    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54f4b6d0_be12_40ae_b3ac_f7acb8e715b9.slice/crio-6e9e71f7b746e7671e7931ee749e941b1d10eea097fe41e7e99d35005966801d WatchSource:0}: Error finding container 6e9e71f7b746e7671e7931ee749e941b1d10eea097fe41e7e99d35005966801d: Status 404 returned error can't find the container with id 6e9e71f7b746e7671e7931ee749e941b1d10eea097fe41e7e99d35005966801d
Jul 24 16:49:11.250113 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:11.250073    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8lwz8" event={"ID":"54f4b6d0-be12-40ae-b3ac-f7acb8e715b9","Type":"ContainerStarted","Data":"6e9e71f7b746e7671e7931ee749e941b1d10eea097fe41e7e99d35005966801d"}
Jul 24 16:49:13.261743 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:13.261706    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8lwz8" event={"ID":"54f4b6d0-be12-40ae-b3ac-f7acb8e715b9","Type":"ContainerStarted","Data":"530e01ddb0d0ff8d7eacb32669a36872b3be9b1d0d94ea7e6369ef224fbbd796"}
Jul 24 16:49:13.262133 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:13.261797    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8lwz8" event={"ID":"54f4b6d0-be12-40ae-b3ac-f7acb8e715b9","Type":"ContainerStarted","Data":"966a34c068976ccc0368ef52e04a780604997c881aea4d09fc1e8be8b376525a"}
Jul 24 16:49:13.279753 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:13.279702    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8lwz8" podStartSLOduration=373.131308942 podStartE2EDuration="6m14.279686623s" totalImagesPullingTime="1.148377681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:42:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:49:13.277776897 +0000 UTC m=+375.919740804" watchObservedRunningTime="2026-07-24 16:49:13.279686623 +0000 UTC m=+375.921650507"
Jul 24 16:49:13.363644 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:13.363594    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:49:13.464696 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:13.464645    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:49:13.517840 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:13.517748    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:49:14.839024 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:14.838993    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:49:14.893288 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:14.893257    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:49:14.947091 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:14.947057    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:49:18.228141 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:18.228106    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:49:18.364730 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:18.364695    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:49:27.679158 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:27.679080    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8fc9769c4-2cbwj"]
Jul 24 16:49:27.680329 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:27.680281    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e7f3a6f-1da1-4a07-b736-512dfb2afd6a" containerName="registry"
Jul 24 16:49:27.680512 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:27.680500    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e7f3a6f-1da1-4a07-b736-512dfb2afd6a" containerName="registry"
Jul 24 16:49:27.680694 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:27.680681    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e7f3a6f-1da1-4a07-b736-512dfb2afd6a" containerName="registry"
Jul 24 16:49:27.684240 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:27.684216    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8fc9769c4-2cbwj"
Jul 24 16:49:27.692488 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:27.692452    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8fc9769c4-2cbwj"]
Jul 24 16:49:27.765952 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:27.765915    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:49:27.792067 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:27.792031    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a74f6a5-5176-4e77-b3b3-cede482f4441-console-serving-cert\") pod \"console-8fc9769c4-2cbwj\" (UID: \"8a74f6a5-5176-4e77-b3b3-cede482f4441\") " pod="openshift-console/console-8fc9769c4-2cbwj"
Jul 24 16:49:27.792067 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:27.792069    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgfkm\" (UniqueName: \"kubernetes.io/projected/8a74f6a5-5176-4e77-b3b3-cede482f4441-kube-api-access-mgfkm\") pod \"console-8fc9769c4-2cbwj\" (UID: \"8a74f6a5-5176-4e77-b3b3-cede482f4441\") " pod="openshift-console/console-8fc9769c4-2cbwj"
Jul 24 16:49:27.792297 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:27.792090    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a74f6a5-5176-4e77-b3b3-cede482f4441-oauth-serving-cert\") pod \"console-8fc9769c4-2cbwj\" (UID: \"8a74f6a5-5176-4e77-b3b3-cede482f4441\") " pod="openshift-console/console-8fc9769c4-2cbwj"
Jul 24 16:49:27.792297 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:27.792143    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a74f6a5-5176-4e77-b3b3-cede482f4441-trusted-ca-bundle\") pod \"console-8fc9769c4-2cbwj\" (UID: \"8a74f6a5-5176-4e77-b3b3-cede482f4441\") " pod="openshift-console/console-8fc9769c4-2cbwj"
Jul 24 16:49:27.792297 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:27.792163    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a74f6a5-5176-4e77-b3b3-cede482f4441-service-ca\") pod \"console-8fc9769c4-2cbwj\" (UID: \"8a74f6a5-5176-4e77-b3b3-cede482f4441\") " pod="openshift-console/console-8fc9769c4-2cbwj"
Jul 24 16:49:27.792297 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:27.792201    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a74f6a5-5176-4e77-b3b3-cede482f4441-console-oauth-config\") pod \"console-8fc9769c4-2cbwj\" (UID: \"8a74f6a5-5176-4e77-b3b3-cede482f4441\") " pod="openshift-console/console-8fc9769c4-2cbwj"
Jul 24 16:49:27.792297 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:27.792234    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a74f6a5-5176-4e77-b3b3-cede482f4441-console-config\") pod \"console-8fc9769c4-2cbwj\" (UID: \"8a74f6a5-5176-4e77-b3b3-cede482f4441\") " pod="openshift-console/console-8fc9769c4-2cbwj"
Jul 24 16:49:27.863926 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:27.863891    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:49:27.893216 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:27.893181    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a74f6a5-5176-4e77-b3b3-cede482f4441-oauth-serving-cert\") pod \"console-8fc9769c4-2cbwj\" (UID: \"8a74f6a5-5176-4e77-b3b3-cede482f4441\") " pod="openshift-console/console-8fc9769c4-2cbwj"
Jul 24 16:49:27.893361 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:27.893296    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a74f6a5-5176-4e77-b3b3-cede482f4441-trusted-ca-bundle\") pod \"console-8fc9769c4-2cbwj\" (UID: \"8a74f6a5-5176-4e77-b3b3-cede482f4441\") " pod="openshift-console/console-8fc9769c4-2cbwj"
Jul 24 16:49:27.893361 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:27.893317    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a74f6a5-5176-4e77-b3b3-cede482f4441-service-ca\") pod \"console-8fc9769c4-2cbwj\" (UID: \"8a74f6a5-5176-4e77-b3b3-cede482f4441\") " pod="openshift-console/console-8fc9769c4-2cbwj"
Jul 24 16:49:27.893361 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:27.893342    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a74f6a5-5176-4e77-b3b3-cede482f4441-console-oauth-config\") pod \"console-8fc9769c4-2cbwj\" (UID: \"8a74f6a5-5176-4e77-b3b3-cede482f4441\") " pod="openshift-console/console-8fc9769c4-2cbwj"
Jul 24 16:49:27.893481 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:27.893380    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a74f6a5-5176-4e77-b3b3-cede482f4441-console-config\") pod \"console-8fc9769c4-2cbwj\" (UID: \"8a74f6a5-5176-4e77-b3b3-cede482f4441\") " pod="openshift-console/console-8fc9769c4-2cbwj"
Jul 24 16:49:27.893481 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:27.893417    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a74f6a5-5176-4e77-b3b3-cede482f4441-console-serving-cert\") pod \"console-8fc9769c4-2cbwj\" (UID: \"8a74f6a5-5176-4e77-b3b3-cede482f4441\") " pod="openshift-console/console-8fc9769c4-2cbwj"
Jul 24 16:49:27.893481 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:27.893442    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgfkm\" (UniqueName: \"kubernetes.io/projected/8a74f6a5-5176-4e77-b3b3-cede482f4441-kube-api-access-mgfkm\") pod \"console-8fc9769c4-2cbwj\" (UID: \"8a74f6a5-5176-4e77-b3b3-cede482f4441\") " pod="openshift-console/console-8fc9769c4-2cbwj"
Jul 24 16:49:27.893960 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:27.893920    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a74f6a5-5176-4e77-b3b3-cede482f4441-oauth-serving-cert\") pod \"console-8fc9769c4-2cbwj\" (UID: \"8a74f6a5-5176-4e77-b3b3-cede482f4441\") " pod="openshift-console/console-8fc9769c4-2cbwj"
Jul 24 16:49:27.894073 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:27.894053    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a74f6a5-5176-4e77-b3b3-cede482f4441-service-ca\") pod \"console-8fc9769c4-2cbwj\" (UID: \"8a74f6a5-5176-4e77-b3b3-cede482f4441\") " pod="openshift-console/console-8fc9769c4-2cbwj"
Jul 24 16:49:27.894261 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:27.894236    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a74f6a5-5176-4e77-b3b3-cede482f4441-trusted-ca-bundle\") pod \"console-8fc9769c4-2cbwj\" (UID: \"8a74f6a5-5176-4e77-b3b3-cede482f4441\") " pod="openshift-console/console-8fc9769c4-2cbwj"
Jul 24 16:49:27.894261 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:27.894250    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a74f6a5-5176-4e77-b3b3-cede482f4441-console-config\") pod \"console-8fc9769c4-2cbwj\" (UID: \"8a74f6a5-5176-4e77-b3b3-cede482f4441\") " pod="openshift-console/console-8fc9769c4-2cbwj"
Jul 24 16:49:27.895904 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:27.895884    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a74f6a5-5176-4e77-b3b3-cede482f4441-console-oauth-config\") pod \"console-8fc9769c4-2cbwj\" (UID: \"8a74f6a5-5176-4e77-b3b3-cede482f4441\") " pod="openshift-console/console-8fc9769c4-2cbwj"
Jul 24 16:49:27.895982 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:27.895905    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a74f6a5-5176-4e77-b3b3-cede482f4441-console-serving-cert\") pod \"console-8fc9769c4-2cbwj\" (UID: \"8a74f6a5-5176-4e77-b3b3-cede482f4441\") " pod="openshift-console/console-8fc9769c4-2cbwj"
Jul 24 16:49:27.900388 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:27.900368    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgfkm\" (UniqueName: \"kubernetes.io/projected/8a74f6a5-5176-4e77-b3b3-cede482f4441-kube-api-access-mgfkm\") pod \"console-8fc9769c4-2cbwj\" (UID: \"8a74f6a5-5176-4e77-b3b3-cede482f4441\") " pod="openshift-console/console-8fc9769c4-2cbwj"
Jul 24 16:49:27.955706 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:27.955634    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:49:27.995827 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:27.995797    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8fc9769c4-2cbwj"
Jul 24 16:49:28.008166 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:28.008135    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:49:28.130394 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:28.130353    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8fc9769c4-2cbwj"]
Jul 24 16:49:28.133443 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:49:28.133416    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a74f6a5_5176_4e77_b3b3_cede482f4441.slice/crio-dcd2d0bfa2e57f493b91b888211507534dbcf5482c0904c998f758369e8f1254 WatchSource:0}: Error finding container dcd2d0bfa2e57f493b91b888211507534dbcf5482c0904c998f758369e8f1254: Status 404 returned error can't find the container with id dcd2d0bfa2e57f493b91b888211507534dbcf5482c0904c998f758369e8f1254
Jul 24 16:49:28.329978 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:28.329870    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8fc9769c4-2cbwj" event={"ID":"8a74f6a5-5176-4e77-b3b3-cede482f4441","Type":"ContainerStarted","Data":"3a9f1fd6069b895a371d9043a38f788295ccaa705fc86da2b58a7eeed1765cc6"}
Jul 24 16:49:28.329978 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:28.329932    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8fc9769c4-2cbwj" event={"ID":"8a74f6a5-5176-4e77-b3b3-cede482f4441","Type":"ContainerStarted","Data":"dcd2d0bfa2e57f493b91b888211507534dbcf5482c0904c998f758369e8f1254"}
Jul 24 16:49:28.348233 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:28.348182    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8fc9769c4-2cbwj" podStartSLOduration=1.34816798 podStartE2EDuration="1.34816798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:49:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:49:28.346068498 +0000 UTC m=+390.988032383" watchObservedRunningTime="2026-07-24 16:49:28.34816798 +0000 UTC m=+390.990131930"
Jul 24 16:49:28.471486 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:28.471457    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:49:28.967320 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:28.967277    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:49:29.032419 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:29.032389    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:49:37.996831 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:37.996794    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8fc9769c4-2cbwj"
Jul 24 16:49:37.998873 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:37.998851    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8fc9769c4-2cbwj"
Jul 24 16:49:38.004652 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:38.004636    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8fc9769c4-2cbwj"
Jul 24 16:49:38.374126 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:38.374049    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8fc9769c4-2cbwj"
Jul 24 16:49:38.417999 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:38.417961    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84b84cc59-mhbkn"]
Jul 24 16:49:38.520370 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:38.520336    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:49:38.568325 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:38.568297    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:49:38.625390 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:49:38.625307    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:50:03.440189 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:03.440128    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-84b84cc59-mhbkn" podUID="dd24175c-399c-4773-b526-5e5b63a5e019" containerName="console" containerID="cri-o://a9dee69c3fe1d6af3ba91d1366c65a5c9fe4a73038667f4a4a7fa01a55082001" gracePeriod=15
Jul 24 16:50:03.679452 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:03.679422    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84b84cc59-mhbkn"
Jul 24 16:50:03.689826 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:03.689799    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd24175c-399c-4773-b526-5e5b63a5e019-service-ca\") pod \"dd24175c-399c-4773-b526-5e5b63a5e019\" (UID: \"dd24175c-399c-4773-b526-5e5b63a5e019\") "
Jul 24 16:50:03.689987 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:03.689851    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhjwz\" (UniqueName: \"kubernetes.io/projected/dd24175c-399c-4773-b526-5e5b63a5e019-kube-api-access-xhjwz\") pod \"dd24175c-399c-4773-b526-5e5b63a5e019\" (UID: \"dd24175c-399c-4773-b526-5e5b63a5e019\") "
Jul 24 16:50:03.689987 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:03.689894    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd24175c-399c-4773-b526-5e5b63a5e019-console-config\") pod \"dd24175c-399c-4773-b526-5e5b63a5e019\" (UID: \"dd24175c-399c-4773-b526-5e5b63a5e019\") "
Jul 24 16:50:03.689987 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:03.689918    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd24175c-399c-4773-b526-5e5b63a5e019-console-oauth-config\") pod \"dd24175c-399c-4773-b526-5e5b63a5e019\" (UID: \"dd24175c-399c-4773-b526-5e5b63a5e019\") "
Jul 24 16:50:03.689987 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:03.689953    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd24175c-399c-4773-b526-5e5b63a5e019-console-serving-cert\") pod \"dd24175c-399c-4773-b526-5e5b63a5e019\" (UID: \"dd24175c-399c-4773-b526-5e5b63a5e019\") "
Jul 24 16:50:03.690236 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:03.690140    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd24175c-399c-4773-b526-5e5b63a5e019-trusted-ca-bundle\") pod \"dd24175c-399c-4773-b526-5e5b63a5e019\" (UID: \"dd24175c-399c-4773-b526-5e5b63a5e019\") "
Jul 24 16:50:03.690236 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:03.690182    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd24175c-399c-4773-b526-5e5b63a5e019-oauth-serving-cert\") pod \"dd24175c-399c-4773-b526-5e5b63a5e019\" (UID: \"dd24175c-399c-4773-b526-5e5b63a5e019\") "
Jul 24 16:50:03.690342 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:03.690277    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd24175c-399c-4773-b526-5e5b63a5e019-service-ca" (OuterVolumeSpecName: "service-ca") pod "dd24175c-399c-4773-b526-5e5b63a5e019" (UID: "dd24175c-399c-4773-b526-5e5b63a5e019"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 16:50:03.690406 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:03.690382    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd24175c-399c-4773-b526-5e5b63a5e019-console-config" (OuterVolumeSpecName: "console-config") pod "dd24175c-399c-4773-b526-5e5b63a5e019" (UID: "dd24175c-399c-4773-b526-5e5b63a5e019"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 16:50:03.690544 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:03.690479    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd24175c-399c-4773-b526-5e5b63a5e019-console-config\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 16:50:03.690609 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:03.690542    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd24175c-399c-4773-b526-5e5b63a5e019-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "dd24175c-399c-4773-b526-5e5b63a5e019" (UID: "dd24175c-399c-4773-b526-5e5b63a5e019"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 16:50:03.690609 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:03.690551    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd24175c-399c-4773-b526-5e5b63a5e019-service-ca\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 16:50:03.690697 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:03.690619    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd24175c-399c-4773-b526-5e5b63a5e019-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "dd24175c-399c-4773-b526-5e5b63a5e019" (UID: "dd24175c-399c-4773-b526-5e5b63a5e019"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 16:50:03.692248 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:03.692222    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd24175c-399c-4773-b526-5e5b63a5e019-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "dd24175c-399c-4773-b526-5e5b63a5e019" (UID: "dd24175c-399c-4773-b526-5e5b63a5e019"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 16:50:03.692344 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:03.692281    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd24175c-399c-4773-b526-5e5b63a5e019-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "dd24175c-399c-4773-b526-5e5b63a5e019" (UID: "dd24175c-399c-4773-b526-5e5b63a5e019"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 16:50:03.692344 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:03.692320    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd24175c-399c-4773-b526-5e5b63a5e019-kube-api-access-xhjwz" (OuterVolumeSpecName: "kube-api-access-xhjwz") pod "dd24175c-399c-4773-b526-5e5b63a5e019" (UID: "dd24175c-399c-4773-b526-5e5b63a5e019"). InnerVolumeSpecName "kube-api-access-xhjwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 16:50:03.795609 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:03.791561    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhjwz\" (UniqueName: \"kubernetes.io/projected/dd24175c-399c-4773-b526-5e5b63a5e019-kube-api-access-xhjwz\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 16:50:03.795609 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:03.791597    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd24175c-399c-4773-b526-5e5b63a5e019-console-oauth-config\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 16:50:03.795609 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:03.791619    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd24175c-399c-4773-b526-5e5b63a5e019-console-serving-cert\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 16:50:03.795609 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:03.791632    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd24175c-399c-4773-b526-5e5b63a5e019-trusted-ca-bundle\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 16:50:03.795609 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:03.791645    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd24175c-399c-4773-b526-5e5b63a5e019-oauth-serving-cert\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 16:50:04.470674 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:04.470635    2563 generic.go:356] "Generic (PLEG): container finished" podID="dd24175c-399c-4773-b526-5e5b63a5e019" containerID="a9dee69c3fe1d6af3ba91d1366c65a5c9fe4a73038667f4a4a7fa01a55082001" exitCode=0
Jul 24 16:50:04.471087 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:04.470694    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84b84cc59-mhbkn"
Jul 24 16:50:04.471087 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:04.470704    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84b84cc59-mhbkn" event={"ID":"dd24175c-399c-4773-b526-5e5b63a5e019","Type":"ContainerDied","Data":"a9dee69c3fe1d6af3ba91d1366c65a5c9fe4a73038667f4a4a7fa01a55082001"}
Jul 24 16:50:04.471087 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:04.470751    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84b84cc59-mhbkn" event={"ID":"dd24175c-399c-4773-b526-5e5b63a5e019","Type":"ContainerDied","Data":"0fef68f74fa428f4541f6c3a8632a0cd90d05697d9ea6d9dd5b2d6019ea0f3ce"}
Jul 24 16:50:04.471087 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:04.470772    2563 scope.go:117] "RemoveContainer" containerID="a9dee69c3fe1d6af3ba91d1366c65a5c9fe4a73038667f4a4a7fa01a55082001"
Jul 24 16:50:04.479239 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:04.479218    2563 scope.go:117] "RemoveContainer" containerID="a9dee69c3fe1d6af3ba91d1366c65a5c9fe4a73038667f4a4a7fa01a55082001"
Jul 24 16:50:04.479488 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:50:04.479469    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9dee69c3fe1d6af3ba91d1366c65a5c9fe4a73038667f4a4a7fa01a55082001\": container with ID starting with a9dee69c3fe1d6af3ba91d1366c65a5c9fe4a73038667f4a4a7fa01a55082001 not found: ID does not exist" containerID="a9dee69c3fe1d6af3ba91d1366c65a5c9fe4a73038667f4a4a7fa01a55082001"
Jul 24 16:50:04.479534 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:04.479497    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9dee69c3fe1d6af3ba91d1366c65a5c9fe4a73038667f4a4a7fa01a55082001"} err="failed to get container status \"a9dee69c3fe1d6af3ba91d1366c65a5c9fe4a73038667f4a4a7fa01a55082001\": rpc error: code = NotFound desc = could not find container \"a9dee69c3fe1d6af3ba91d1366c65a5c9fe4a73038667f4a4a7fa01a55082001\": container with ID starting with a9dee69c3fe1d6af3ba91d1366c65a5c9fe4a73038667f4a4a7fa01a55082001 not found: ID does not exist"
Jul 24 16:50:04.494173 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:04.494148    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84b84cc59-mhbkn"]
Jul 24 16:50:04.501823 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:04.501805    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84b84cc59-mhbkn"]
Jul 24 16:50:06.001505 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:06.001472    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd24175c-399c-4773-b526-5e5b63a5e019" path="/var/lib/kubelet/pods/dd24175c-399c-4773-b526-5e5b63a5e019/volumes"
Jul 24 16:50:24.667274 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:24.667154    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:50:31.209978 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:31.209902    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:50:31.311800 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:31.311757    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:50:33.481851 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:33.481819    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:50:33.584061 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:33.584027    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:50:42.274572 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:42.274538    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-848847dcf-5lw89"]
Jul 24 16:50:42.275004 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:42.274856    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd24175c-399c-4773-b526-5e5b63a5e019" containerName="console"
Jul 24 16:50:42.275004 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:42.274868    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd24175c-399c-4773-b526-5e5b63a5e019" containerName="console"
Jul 24 16:50:42.275004 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:42.274916    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd24175c-399c-4773-b526-5e5b63a5e019" containerName="console"
Jul 24 16:50:42.277973 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:42.277921    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-848847dcf-5lw89"
Jul 24 16:50:42.296178 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:42.296149    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-848847dcf-5lw89"]
Jul 24 16:50:42.335998 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:42.335967    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:50:42.379860 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:42.379826    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cb0e346b-a668-47f7-b80e-982673ba98a0-service-ca\") pod \"console-848847dcf-5lw89\" (UID: \"cb0e346b-a668-47f7-b80e-982673ba98a0\") " pod="openshift-console/console-848847dcf-5lw89"
Jul 24 16:50:42.379860 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:42.379862    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cb0e346b-a668-47f7-b80e-982673ba98a0-console-config\") pod \"console-848847dcf-5lw89\" (UID: \"cb0e346b-a668-47f7-b80e-982673ba98a0\") " pod="openshift-console/console-848847dcf-5lw89"
Jul 24 16:50:42.380096 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:42.379907    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cb0e346b-a668-47f7-b80e-982673ba98a0-console-oauth-config\") pod \"console-848847dcf-5lw89\" (UID: \"cb0e346b-a668-47f7-b80e-982673ba98a0\") " pod="openshift-console/console-848847dcf-5lw89"
Jul 24 16:50:42.380096 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:42.379982    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cb0e346b-a668-47f7-b80e-982673ba98a0-oauth-serving-cert\") pod \"console-848847dcf-5lw89\" (UID: \"cb0e346b-a668-47f7-b80e-982673ba98a0\") " pod="openshift-console/console-848847dcf-5lw89"
Jul 24 16:50:42.380096 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:42.380001    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rnrj\" (UniqueName: \"kubernetes.io/projected/cb0e346b-a668-47f7-b80e-982673ba98a0-kube-api-access-5rnrj\") pod \"console-848847dcf-5lw89\" (UID: \"cb0e346b-a668-47f7-b80e-982673ba98a0\") " pod="openshift-console/console-848847dcf-5lw89"
Jul 24 16:50:42.380096 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:42.380023    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb0e346b-a668-47f7-b80e-982673ba98a0-console-serving-cert\") pod \"console-848847dcf-5lw89\" (UID: \"cb0e346b-a668-47f7-b80e-982673ba98a0\") " pod="openshift-console/console-848847dcf-5lw89"
Jul 24 16:50:42.380096 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:42.380041    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb0e346b-a668-47f7-b80e-982673ba98a0-trusted-ca-bundle\") pod \"console-848847dcf-5lw89\" (UID: \"cb0e346b-a668-47f7-b80e-982673ba98a0\") " pod="openshift-console/console-848847dcf-5lw89"
Jul 24 16:50:42.400195 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:42.400163    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:50:42.481264 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:42.481219    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cb0e346b-a668-47f7-b80e-982673ba98a0-oauth-serving-cert\") pod \"console-848847dcf-5lw89\" (UID: \"cb0e346b-a668-47f7-b80e-982673ba98a0\") " pod="openshift-console/console-848847dcf-5lw89"
Jul 24 16:50:42.481264 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:42.481267    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rnrj\" (UniqueName: \"kubernetes.io/projected/cb0e346b-a668-47f7-b80e-982673ba98a0-kube-api-access-5rnrj\") pod \"console-848847dcf-5lw89\" (UID: \"cb0e346b-a668-47f7-b80e-982673ba98a0\") " pod="openshift-console/console-848847dcf-5lw89"
Jul 24 16:50:42.481500 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:42.481303    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb0e346b-a668-47f7-b80e-982673ba98a0-console-serving-cert\") pod \"console-848847dcf-5lw89\" (UID: \"cb0e346b-a668-47f7-b80e-982673ba98a0\") " pod="openshift-console/console-848847dcf-5lw89"
Jul 24 16:50:42.481500 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:42.481330    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb0e346b-a668-47f7-b80e-982673ba98a0-trusted-ca-bundle\") pod \"console-848847dcf-5lw89\" (UID: \"cb0e346b-a668-47f7-b80e-982673ba98a0\") " pod="openshift-console/console-848847dcf-5lw89"
Jul 24 16:50:42.481500 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:42.481364    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cb0e346b-a668-47f7-b80e-982673ba98a0-service-ca\") pod \"console-848847dcf-5lw89\" (UID: \"cb0e346b-a668-47f7-b80e-982673ba98a0\") " pod="openshift-console/console-848847dcf-5lw89"
Jul 24 16:50:42.481500 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:42.481392    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cb0e346b-a668-47f7-b80e-982673ba98a0-console-config\") pod \"console-848847dcf-5lw89\" (UID: \"cb0e346b-a668-47f7-b80e-982673ba98a0\") " pod="openshift-console/console-848847dcf-5lw89"
Jul 24 16:50:42.481646 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:42.481590    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cb0e346b-a668-47f7-b80e-982673ba98a0-console-oauth-config\") pod \"console-848847dcf-5lw89\" (UID: \"cb0e346b-a668-47f7-b80e-982673ba98a0\") " pod="openshift-console/console-848847dcf-5lw89"
Jul 24 16:50:42.482065 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:42.482036    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cb0e346b-a668-47f7-b80e-982673ba98a0-oauth-serving-cert\") pod \"console-848847dcf-5lw89\" (UID: \"cb0e346b-a668-47f7-b80e-982673ba98a0\") " pod="openshift-console/console-848847dcf-5lw89"
Jul 24 16:50:42.482065 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:42.482057    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cb0e346b-a668-47f7-b80e-982673ba98a0-service-ca\") pod \"console-848847dcf-5lw89\" (UID: \"cb0e346b-a668-47f7-b80e-982673ba98a0\") " pod="openshift-console/console-848847dcf-5lw89"
Jul 24 16:50:42.482314 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:42.482295    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cb0e346b-a668-47f7-b80e-982673ba98a0-console-config\") pod \"console-848847dcf-5lw89\" (UID: \"cb0e346b-a668-47f7-b80e-982673ba98a0\") " pod="openshift-console/console-848847dcf-5lw89"
Jul 24 16:50:42.482391 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:42.482375    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb0e346b-a668-47f7-b80e-982673ba98a0-trusted-ca-bundle\") pod \"console-848847dcf-5lw89\" (UID: \"cb0e346b-a668-47f7-b80e-982673ba98a0\") " pod="openshift-console/console-848847dcf-5lw89"
Jul 24 16:50:42.484266 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:42.484241    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cb0e346b-a668-47f7-b80e-982673ba98a0-console-oauth-config\") pod \"console-848847dcf-5lw89\" (UID: \"cb0e346b-a668-47f7-b80e-982673ba98a0\") " pod="openshift-console/console-848847dcf-5lw89"
Jul 24 16:50:42.484347 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:42.484330    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb0e346b-a668-47f7-b80e-982673ba98a0-console-serving-cert\") pod \"console-848847dcf-5lw89\" (UID: \"cb0e346b-a668-47f7-b80e-982673ba98a0\") " pod="openshift-console/console-848847dcf-5lw89"
Jul 24 16:50:42.490054 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:42.490027    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rnrj\" (UniqueName: \"kubernetes.io/projected/cb0e346b-a668-47f7-b80e-982673ba98a0-kube-api-access-5rnrj\") pod \"console-848847dcf-5lw89\" (UID: \"cb0e346b-a668-47f7-b80e-982673ba98a0\") " pod="openshift-console/console-848847dcf-5lw89"
Jul 24 16:50:42.588307 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:42.588216    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-848847dcf-5lw89"
Jul 24 16:50:42.723820 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:42.723795    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-848847dcf-5lw89"]
Jul 24 16:50:42.726581 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:50:42.726552    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb0e346b_a668_47f7_b80e_982673ba98a0.slice/crio-db47a294cbde609c907c9d0e0308cfdf59f6ee4c5afc7db5d8f0324327da7ad6 WatchSource:0}: Error finding container db47a294cbde609c907c9d0e0308cfdf59f6ee4c5afc7db5d8f0324327da7ad6: Status 404 returned error can't find the container with id db47a294cbde609c907c9d0e0308cfdf59f6ee4c5afc7db5d8f0324327da7ad6
Jul 24 16:50:43.617429 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:43.617393    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-848847dcf-5lw89" event={"ID":"cb0e346b-a668-47f7-b80e-982673ba98a0","Type":"ContainerStarted","Data":"817f477236ff1328d97fd335940d979b75b5a8a7ccafcd986d9af189f0caba0b"}
Jul 24 16:50:43.617871 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:43.617440    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-848847dcf-5lw89" event={"ID":"cb0e346b-a668-47f7-b80e-982673ba98a0","Type":"ContainerStarted","Data":"db47a294cbde609c907c9d0e0308cfdf59f6ee4c5afc7db5d8f0324327da7ad6"}
Jul 24 16:50:43.634756 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:43.634700    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-848847dcf-5lw89" podStartSLOduration=1.634684255 podStartE2EDuration="1.634684255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:50:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:50:43.633679473 +0000 UTC m=+466.275643376" watchObservedRunningTime="2026-07-24 16:50:43.634684255 +0000 UTC m=+466.276648138"
Jul 24 16:50:45.375826 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:45.375790    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:50:45.427974 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:45.427945    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:50:52.589125 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:52.589085    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-848847dcf-5lw89"
Jul 24 16:50:52.589520 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:52.589147    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-848847dcf-5lw89"
Jul 24 16:50:52.593978 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:52.593930    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-848847dcf-5lw89"
Jul 24 16:50:52.659648 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:52.659619    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-848847dcf-5lw89"
Jul 24 16:50:52.705751 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:52.705718    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8fc9769c4-2cbwj"]
Jul 24 16:50:52.822195 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:52.822166    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:50:52.942016 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:52.941986    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:50:53.043098 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:53.043064    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:50:59.294678 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:59.294646    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:50:59.397130 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:50:59.397094    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:51:17.728051 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:17.727991    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8fc9769c4-2cbwj" podUID="8a74f6a5-5176-4e77-b3b3-cede482f4441" containerName="console" containerID="cri-o://3a9f1fd6069b895a371d9043a38f788295ccaa705fc86da2b58a7eeed1765cc6" gracePeriod=15
Jul 24 16:51:17.970630 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:17.970606    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8fc9769c4-2cbwj"
Jul 24 16:51:17.977884 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:17.977864    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgfkm\" (UniqueName: \"kubernetes.io/projected/8a74f6a5-5176-4e77-b3b3-cede482f4441-kube-api-access-mgfkm\") pod \"8a74f6a5-5176-4e77-b3b3-cede482f4441\" (UID: \"8a74f6a5-5176-4e77-b3b3-cede482f4441\") "
Jul 24 16:51:17.978010 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:17.977912    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a74f6a5-5176-4e77-b3b3-cede482f4441-console-oauth-config\") pod \"8a74f6a5-5176-4e77-b3b3-cede482f4441\" (UID: \"8a74f6a5-5176-4e77-b3b3-cede482f4441\") "
Jul 24 16:51:17.978010 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:17.977962    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a74f6a5-5176-4e77-b3b3-cede482f4441-oauth-serving-cert\") pod \"8a74f6a5-5176-4e77-b3b3-cede482f4441\" (UID: \"8a74f6a5-5176-4e77-b3b3-cede482f4441\") "
Jul 24 16:51:17.978010 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:17.977991    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a74f6a5-5176-4e77-b3b3-cede482f4441-console-config\") pod \"8a74f6a5-5176-4e77-b3b3-cede482f4441\" (UID: \"8a74f6a5-5176-4e77-b3b3-cede482f4441\") "
Jul 24 16:51:17.978191 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:17.978022    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a74f6a5-5176-4e77-b3b3-cede482f4441-console-serving-cert\") pod \"8a74f6a5-5176-4e77-b3b3-cede482f4441\" (UID: \"8a74f6a5-5176-4e77-b3b3-cede482f4441\") "
Jul 24 16:51:17.978191 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:17.978052    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a74f6a5-5176-4e77-b3b3-cede482f4441-trusted-ca-bundle\") pod \"8a74f6a5-5176-4e77-b3b3-cede482f4441\" (UID: \"8a74f6a5-5176-4e77-b3b3-cede482f4441\") "
Jul 24 16:51:17.978191 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:17.978087    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a74f6a5-5176-4e77-b3b3-cede482f4441-service-ca\") pod \"8a74f6a5-5176-4e77-b3b3-cede482f4441\" (UID: \"8a74f6a5-5176-4e77-b3b3-cede482f4441\") "
Jul 24 16:51:17.978408 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:17.978382    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a74f6a5-5176-4e77-b3b3-cede482f4441-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8a74f6a5-5176-4e77-b3b3-cede482f4441" (UID: "8a74f6a5-5176-4e77-b3b3-cede482f4441"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 16:51:17.978486 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:17.978411    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a74f6a5-5176-4e77-b3b3-cede482f4441-console-config" (OuterVolumeSpecName: "console-config") pod "8a74f6a5-5176-4e77-b3b3-cede482f4441" (UID: "8a74f6a5-5176-4e77-b3b3-cede482f4441"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 16:51:17.978486 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:17.978434    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a74f6a5-5176-4e77-b3b3-cede482f4441-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8a74f6a5-5176-4e77-b3b3-cede482f4441" (UID: "8a74f6a5-5176-4e77-b3b3-cede482f4441"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 16:51:17.978647 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:17.978617    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a74f6a5-5176-4e77-b3b3-cede482f4441-service-ca" (OuterVolumeSpecName: "service-ca") pod "8a74f6a5-5176-4e77-b3b3-cede482f4441" (UID: "8a74f6a5-5176-4e77-b3b3-cede482f4441"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 16:51:17.980284 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:17.980257    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a74f6a5-5176-4e77-b3b3-cede482f4441-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8a74f6a5-5176-4e77-b3b3-cede482f4441" (UID: "8a74f6a5-5176-4e77-b3b3-cede482f4441"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 16:51:17.980492 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:17.980468    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a74f6a5-5176-4e77-b3b3-cede482f4441-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8a74f6a5-5176-4e77-b3b3-cede482f4441" (UID: "8a74f6a5-5176-4e77-b3b3-cede482f4441"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 16:51:17.980492 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:17.980479    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a74f6a5-5176-4e77-b3b3-cede482f4441-kube-api-access-mgfkm" (OuterVolumeSpecName: "kube-api-access-mgfkm") pod "8a74f6a5-5176-4e77-b3b3-cede482f4441" (UID: "8a74f6a5-5176-4e77-b3b3-cede482f4441"). InnerVolumeSpecName "kube-api-access-mgfkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 16:51:18.078851 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:18.078821    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a74f6a5-5176-4e77-b3b3-cede482f4441-service-ca\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 16:51:18.078851 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:18.078847    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgfkm\" (UniqueName: \"kubernetes.io/projected/8a74f6a5-5176-4e77-b3b3-cede482f4441-kube-api-access-mgfkm\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 16:51:18.078851 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:18.078857    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a74f6a5-5176-4e77-b3b3-cede482f4441-console-oauth-config\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 16:51:18.079096 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:18.078866    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a74f6a5-5176-4e77-b3b3-cede482f4441-oauth-serving-cert\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 16:51:18.079096 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:18.078875    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a74f6a5-5176-4e77-b3b3-cede482f4441-console-config\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 16:51:18.079096 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:18.078883    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a74f6a5-5176-4e77-b3b3-cede482f4441-console-serving-cert\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 16:51:18.079096 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:18.078892    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a74f6a5-5176-4e77-b3b3-cede482f4441-trusted-ca-bundle\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 16:51:18.754852 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:18.754814    2563 generic.go:356] "Generic (PLEG): container finished" podID="8a74f6a5-5176-4e77-b3b3-cede482f4441" containerID="3a9f1fd6069b895a371d9043a38f788295ccaa705fc86da2b58a7eeed1765cc6" exitCode=0
Jul 24 16:51:18.755274 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:18.754863    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8fc9769c4-2cbwj"
Jul 24 16:51:18.755274 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:18.754903    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8fc9769c4-2cbwj" event={"ID":"8a74f6a5-5176-4e77-b3b3-cede482f4441","Type":"ContainerDied","Data":"3a9f1fd6069b895a371d9043a38f788295ccaa705fc86da2b58a7eeed1765cc6"}
Jul 24 16:51:18.755274 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:18.754979    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8fc9769c4-2cbwj" event={"ID":"8a74f6a5-5176-4e77-b3b3-cede482f4441","Type":"ContainerDied","Data":"dcd2d0bfa2e57f493b91b888211507534dbcf5482c0904c998f758369e8f1254"}
Jul 24 16:51:18.755274 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:18.755006    2563 scope.go:117] "RemoveContainer" containerID="3a9f1fd6069b895a371d9043a38f788295ccaa705fc86da2b58a7eeed1765cc6"
Jul 24 16:51:18.763209 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:18.763196    2563 scope.go:117] "RemoveContainer" containerID="3a9f1fd6069b895a371d9043a38f788295ccaa705fc86da2b58a7eeed1765cc6"
Jul 24 16:51:18.763453 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:51:18.763436    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a9f1fd6069b895a371d9043a38f788295ccaa705fc86da2b58a7eeed1765cc6\": container with ID starting with 3a9f1fd6069b895a371d9043a38f788295ccaa705fc86da2b58a7eeed1765cc6 not found: ID does not exist" containerID="3a9f1fd6069b895a371d9043a38f788295ccaa705fc86da2b58a7eeed1765cc6"
Jul 24 16:51:18.763502 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:18.763461    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a9f1fd6069b895a371d9043a38f788295ccaa705fc86da2b58a7eeed1765cc6"} err="failed to get container status \"3a9f1fd6069b895a371d9043a38f788295ccaa705fc86da2b58a7eeed1765cc6\": rpc error: code = NotFound desc = could not find container \"3a9f1fd6069b895a371d9043a38f788295ccaa705fc86da2b58a7eeed1765cc6\": container with ID starting with 3a9f1fd6069b895a371d9043a38f788295ccaa705fc86da2b58a7eeed1765cc6 not found: ID does not exist"
Jul 24 16:51:18.771419 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:18.771397    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8fc9769c4-2cbwj"]
Jul 24 16:51:18.775760 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:18.775741    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8fc9769c4-2cbwj"]
Jul 24 16:51:20.006323 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:20.006289    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a74f6a5-5176-4e77-b3b3-cede482f4441" path="/var/lib/kubelet/pods/8a74f6a5-5176-4e77-b3b3-cede482f4441/volumes"
Jul 24 16:51:33.521649 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:33.521606    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:51:33.615868 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:33.615833    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:51:34.285360 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:34.285326    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p9c2g"]
Jul 24 16:51:34.285655 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:34.285643    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a74f6a5-5176-4e77-b3b3-cede482f4441" containerName="console"
Jul 24 16:51:34.285697 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:34.285657    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a74f6a5-5176-4e77-b3b3-cede482f4441" containerName="console"
Jul 24 16:51:34.285734 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:34.285708    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a74f6a5-5176-4e77-b3b3-cede482f4441" containerName="console"
Jul 24 16:51:34.288850 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:34.288830    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p9c2g"
Jul 24 16:51:34.291378 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:34.291362    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-kmbf5\""
Jul 24 16:51:34.291378 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:34.291371    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 24 16:51:34.292285 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:34.292270    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 24 16:51:34.298261 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:34.298242    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p9c2g"]
Jul 24 16:51:34.403831 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:34.403798    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b468g\" (UniqueName: \"kubernetes.io/projected/6677e22e-c5cc-4e1f-8202-e8c2aa69a1db-kube-api-access-b468g\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p9c2g\" (UID: \"6677e22e-c5cc-4e1f-8202-e8c2aa69a1db\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p9c2g"
Jul 24 16:51:34.403831 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:34.403834    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6677e22e-c5cc-4e1f-8202-e8c2aa69a1db-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p9c2g\" (UID: \"6677e22e-c5cc-4e1f-8202-e8c2aa69a1db\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p9c2g"
Jul 24 16:51:34.404119 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:34.403864    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6677e22e-c5cc-4e1f-8202-e8c2aa69a1db-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p9c2g\" (UID: \"6677e22e-c5cc-4e1f-8202-e8c2aa69a1db\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p9c2g"
Jul 24 16:51:34.505176 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:34.505142    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b468g\" (UniqueName: \"kubernetes.io/projected/6677e22e-c5cc-4e1f-8202-e8c2aa69a1db-kube-api-access-b468g\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p9c2g\" (UID: \"6677e22e-c5cc-4e1f-8202-e8c2aa69a1db\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p9c2g"
Jul 24 16:51:34.505176 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:34.505180    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6677e22e-c5cc-4e1f-8202-e8c2aa69a1db-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p9c2g\" (UID: \"6677e22e-c5cc-4e1f-8202-e8c2aa69a1db\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p9c2g"
Jul 24 16:51:34.505369 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:34.505206    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6677e22e-c5cc-4e1f-8202-e8c2aa69a1db-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p9c2g\" (UID: \"6677e22e-c5cc-4e1f-8202-e8c2aa69a1db\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p9c2g"
Jul 24 16:51:34.505541 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:34.505527    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6677e22e-c5cc-4e1f-8202-e8c2aa69a1db-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p9c2g\" (UID: \"6677e22e-c5cc-4e1f-8202-e8c2aa69a1db\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p9c2g"
Jul 24 16:51:34.505578 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:34.505561    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6677e22e-c5cc-4e1f-8202-e8c2aa69a1db-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p9c2g\" (UID: \"6677e22e-c5cc-4e1f-8202-e8c2aa69a1db\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p9c2g"
Jul 24 16:51:34.513850 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:34.513824    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b468g\" (UniqueName: \"kubernetes.io/projected/6677e22e-c5cc-4e1f-8202-e8c2aa69a1db-kube-api-access-b468g\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p9c2g\" (UID: \"6677e22e-c5cc-4e1f-8202-e8c2aa69a1db\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p9c2g"
Jul 24 16:51:34.597575 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:34.597495    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p9c2g"
Jul 24 16:51:34.723689 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:34.723635    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p9c2g"]
Jul 24 16:51:34.726017 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:51:34.725996    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6677e22e_c5cc_4e1f_8202_e8c2aa69a1db.slice/crio-60516d0c80758ccec1338a25f10d26a7eb94d4fd980e58d960e840218bf9669a WatchSource:0}: Error finding container 60516d0c80758ccec1338a25f10d26a7eb94d4fd980e58d960e840218bf9669a: Status 404 returned error can't find the container with id 60516d0c80758ccec1338a25f10d26a7eb94d4fd980e58d960e840218bf9669a
Jul 24 16:51:34.815266 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:34.815238    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p9c2g" event={"ID":"6677e22e-c5cc-4e1f-8202-e8c2aa69a1db","Type":"ContainerStarted","Data":"60516d0c80758ccec1338a25f10d26a7eb94d4fd980e58d960e840218bf9669a"}
Jul 24 16:51:41.844380 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:41.844343    2563 generic.go:356] "Generic (PLEG): container finished" podID="6677e22e-c5cc-4e1f-8202-e8c2aa69a1db" containerID="b9a1747c7914b0d62e34571d9f6d7784d1ee75770aea149f20c466c625321d87" exitCode=0
Jul 24 16:51:41.844758 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:41.844426    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p9c2g" event={"ID":"6677e22e-c5cc-4e1f-8202-e8c2aa69a1db","Type":"ContainerDied","Data":"b9a1747c7914b0d62e34571d9f6d7784d1ee75770aea149f20c466c625321d87"}
Jul 24 16:51:45.860634 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:45.860599    2563 generic.go:356] "Generic (PLEG): container finished" podID="6677e22e-c5cc-4e1f-8202-e8c2aa69a1db" containerID="21952d5c2afc8a7f50d87c3702c19915d3eb1f02eec3b508f5f1528498894d16" exitCode=0
Jul 24 16:51:45.861140 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:45.860657    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p9c2g" event={"ID":"6677e22e-c5cc-4e1f-8202-e8c2aa69a1db","Type":"ContainerDied","Data":"21952d5c2afc8a7f50d87c3702c19915d3eb1f02eec3b508f5f1528498894d16"}
Jul 24 16:51:53.592002 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:53.591977    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:51:53.695623 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:53.695572    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:51:53.895099 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:53.895060    2563 generic.go:356] "Generic (PLEG): container finished" podID="6677e22e-c5cc-4e1f-8202-e8c2aa69a1db" containerID="18280c1cb4bb34265e4cb906be80514061af5688e00a35349124c9406cd7d51a" exitCode=0
Jul 24 16:51:53.895256 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:53.895129    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p9c2g" event={"ID":"6677e22e-c5cc-4e1f-8202-e8c2aa69a1db","Type":"ContainerDied","Data":"18280c1cb4bb34265e4cb906be80514061af5688e00a35349124c9406cd7d51a"}
Jul 24 16:51:55.023561 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:55.023534    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p9c2g"
Jul 24 16:51:55.194335 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:55.194250    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6677e22e-c5cc-4e1f-8202-e8c2aa69a1db-bundle\") pod \"6677e22e-c5cc-4e1f-8202-e8c2aa69a1db\" (UID: \"6677e22e-c5cc-4e1f-8202-e8c2aa69a1db\") "
Jul 24 16:51:55.194335 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:55.194304    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6677e22e-c5cc-4e1f-8202-e8c2aa69a1db-util\") pod \"6677e22e-c5cc-4e1f-8202-e8c2aa69a1db\" (UID: \"6677e22e-c5cc-4e1f-8202-e8c2aa69a1db\") "
Jul 24 16:51:55.194565 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:55.194361    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b468g\" (UniqueName: \"kubernetes.io/projected/6677e22e-c5cc-4e1f-8202-e8c2aa69a1db-kube-api-access-b468g\") pod \"6677e22e-c5cc-4e1f-8202-e8c2aa69a1db\" (UID: \"6677e22e-c5cc-4e1f-8202-e8c2aa69a1db\") "
Jul 24 16:51:55.194924 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:55.194898    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6677e22e-c5cc-4e1f-8202-e8c2aa69a1db-bundle" (OuterVolumeSpecName: "bundle") pod "6677e22e-c5cc-4e1f-8202-e8c2aa69a1db" (UID: "6677e22e-c5cc-4e1f-8202-e8c2aa69a1db"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:51:55.196555 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:55.196531    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6677e22e-c5cc-4e1f-8202-e8c2aa69a1db-kube-api-access-b468g" (OuterVolumeSpecName: "kube-api-access-b468g") pod "6677e22e-c5cc-4e1f-8202-e8c2aa69a1db" (UID: "6677e22e-c5cc-4e1f-8202-e8c2aa69a1db"). InnerVolumeSpecName "kube-api-access-b468g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 16:51:55.200947 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:55.200913    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6677e22e-c5cc-4e1f-8202-e8c2aa69a1db-util" (OuterVolumeSpecName: "util") pod "6677e22e-c5cc-4e1f-8202-e8c2aa69a1db" (UID: "6677e22e-c5cc-4e1f-8202-e8c2aa69a1db"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:51:55.295703 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:55.295668    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6677e22e-c5cc-4e1f-8202-e8c2aa69a1db-util\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 16:51:55.295703 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:55.295702    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b468g\" (UniqueName: \"kubernetes.io/projected/6677e22e-c5cc-4e1f-8202-e8c2aa69a1db-kube-api-access-b468g\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 16:51:55.295891 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:55.295717    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6677e22e-c5cc-4e1f-8202-e8c2aa69a1db-bundle\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 16:51:55.904654 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:55.904625    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p9c2g"
Jul 24 16:51:55.904654 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:55.904639    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p9c2g" event={"ID":"6677e22e-c5cc-4e1f-8202-e8c2aa69a1db","Type":"ContainerDied","Data":"60516d0c80758ccec1338a25f10d26a7eb94d4fd980e58d960e840218bf9669a"}
Jul 24 16:51:55.904870 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:51:55.904704    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60516d0c80758ccec1338a25f10d26a7eb94d4fd980e58d960e840218bf9669a"
Jul 24 16:52:00.450679 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:00.450646    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:52:01.286229 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:01.286198    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:52:04.272173 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:04.272144    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:52:04.375911 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:04.375876    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:52:06.488245 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:06.488211    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:52:06.593158 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:06.593116    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:52:06.853033 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:06.852918    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:52:06.974918 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:06.974885    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:52:08.674957 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:08.674906    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:52:09.082653 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:09.082554    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:52:10.876168 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:10.876136    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:52:10.938181 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:10.938150    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:52:13.618265 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:13.618229    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-mkvf6"]
Jul 24 16:52:13.618646 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:13.618551    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6677e22e-c5cc-4e1f-8202-e8c2aa69a1db" containerName="pull"
Jul 24 16:52:13.618646 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:13.618563    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6677e22e-c5cc-4e1f-8202-e8c2aa69a1db" containerName="pull"
Jul 24 16:52:13.618646 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:13.618579    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6677e22e-c5cc-4e1f-8202-e8c2aa69a1db" containerName="util"
Jul 24 16:52:13.618646 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:13.618584    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6677e22e-c5cc-4e1f-8202-e8c2aa69a1db" containerName="util"
Jul 24 16:52:13.618646 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:13.618595    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6677e22e-c5cc-4e1f-8202-e8c2aa69a1db" containerName="extract"
Jul 24 16:52:13.618646 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:13.618602    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6677e22e-c5cc-4e1f-8202-e8c2aa69a1db" containerName="extract"
Jul 24 16:52:13.618646 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:13.618645    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6677e22e-c5cc-4e1f-8202-e8c2aa69a1db" containerName="extract"
Jul 24 16:52:13.621787 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:13.621768    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-mkvf6"
Jul 24 16:52:13.634197 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:13.634164    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 24 16:52:13.634197 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:13.634190    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 24 16:52:13.634366 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:13.634255    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-4jnkv\""
Jul 24 16:52:13.649041 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:13.649014    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab64551e-e082-49c4-8fd3-ac192ade69ea-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-mkvf6\" (UID: \"ab64551e-e082-49c4-8fd3-ac192ade69ea\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mkvf6"
Jul 24 16:52:13.649125 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:13.649062    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fvk9\" (UniqueName: \"kubernetes.io/projected/ab64551e-e082-49c4-8fd3-ac192ade69ea-kube-api-access-5fvk9\") pod \"cert-manager-webhook-6486ffd997-mkvf6\" (UID: \"ab64551e-e082-49c4-8fd3-ac192ade69ea\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mkvf6"
Jul 24 16:52:13.652978 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:13.652951    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-mkvf6"]
Jul 24 16:52:13.749656 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:13.749618    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab64551e-e082-49c4-8fd3-ac192ade69ea-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-mkvf6\" (UID: \"ab64551e-e082-49c4-8fd3-ac192ade69ea\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mkvf6"
Jul 24 16:52:13.749827 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:13.749668    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fvk9\" (UniqueName: \"kubernetes.io/projected/ab64551e-e082-49c4-8fd3-ac192ade69ea-kube-api-access-5fvk9\") pod \"cert-manager-webhook-6486ffd997-mkvf6\" (UID: \"ab64551e-e082-49c4-8fd3-ac192ade69ea\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mkvf6"
Jul 24 16:52:13.758072 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:13.758045    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab64551e-e082-49c4-8fd3-ac192ade69ea-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-mkvf6\" (UID: \"ab64551e-e082-49c4-8fd3-ac192ade69ea\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mkvf6"
Jul 24 16:52:13.758249 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:13.758230    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fvk9\" (UniqueName: \"kubernetes.io/projected/ab64551e-e082-49c4-8fd3-ac192ade69ea-kube-api-access-5fvk9\") pod \"cert-manager-webhook-6486ffd997-mkvf6\" (UID: \"ab64551e-e082-49c4-8fd3-ac192ade69ea\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mkvf6"
Jul 24 16:52:13.939699 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:13.939663    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-mkvf6"
Jul 24 16:52:14.068542 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:14.068511    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-mkvf6"]
Jul 24 16:52:14.071752 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:52:14.071722    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab64551e_e082_49c4_8fd3_ac192ade69ea.slice/crio-a6b51b919fd800e6cc7f6cf0dca23124c301596464d9ec78aa11779563c3332f WatchSource:0}: Error finding container a6b51b919fd800e6cc7f6cf0dca23124c301596464d9ec78aa11779563c3332f: Status 404 returned error can't find the container with id a6b51b919fd800e6cc7f6cf0dca23124c301596464d9ec78aa11779563c3332f
Jul 24 16:52:14.984574 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:14.984530    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-mkvf6" event={"ID":"ab64551e-e082-49c4-8fd3-ac192ade69ea","Type":"ContainerStarted","Data":"a6b51b919fd800e6cc7f6cf0dca23124c301596464d9ec78aa11779563c3332f"}
Jul 24 16:52:16.648996 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:16.648965    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:52:16.993767 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:16.993670    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-mkvf6" event={"ID":"ab64551e-e082-49c4-8fd3-ac192ade69ea","Type":"ContainerStarted","Data":"59aab3f9b4d33cc3f842dc5e60367ee6f071f1b2ceaab29ebd9b78228dfca9da"}
Jul 24 16:52:16.993767 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:16.993743    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-mkvf6"
Jul 24 16:52:17.008050 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:17.007985    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-mkvf6" podStartSLOduration=1.4298685930000001 podStartE2EDuration="4.007965593s" totalImagesPullingTime="2.578097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:52:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:52:17.007887425 +0000 UTC m=+559.649851310" watchObservedRunningTime="2026-07-24 16:52:17.007965593 +0000 UTC m=+559.649929478"
Jul 24 16:52:18.816125 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:18.816092    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-vbj2p"]
Jul 24 16:52:18.818188 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:18.818167    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vbj2p"
Jul 24 16:52:18.820830 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:18.820811    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-zb5dq\""
Jul 24 16:52:18.828482 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:18.828460    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-vbj2p"]
Jul 24 16:52:18.890830 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:18.890793    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwddk\" (UniqueName: \"kubernetes.io/projected/37e493ac-c476-4852-9c77-f0c2f62cde09-kube-api-access-nwddk\") pod \"cert-manager-cainjector-559cdd8c87-vbj2p\" (UID: \"37e493ac-c476-4852-9c77-f0c2f62cde09\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vbj2p"
Jul 24 16:52:18.891028 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:18.890837    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/37e493ac-c476-4852-9c77-f0c2f62cde09-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-vbj2p\" (UID: \"37e493ac-c476-4852-9c77-f0c2f62cde09\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vbj2p"
Jul 24 16:52:18.992208 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:18.992175    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwddk\" (UniqueName: \"kubernetes.io/projected/37e493ac-c476-4852-9c77-f0c2f62cde09-kube-api-access-nwddk\") pod \"cert-manager-cainjector-559cdd8c87-vbj2p\" (UID: \"37e493ac-c476-4852-9c77-f0c2f62cde09\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vbj2p"
Jul 24 16:52:18.992374 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:18.992214    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/37e493ac-c476-4852-9c77-f0c2f62cde09-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-vbj2p\" (UID: \"37e493ac-c476-4852-9c77-f0c2f62cde09\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vbj2p"
Jul 24 16:52:19.000846 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:19.000818    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwddk\" (UniqueName: \"kubernetes.io/projected/37e493ac-c476-4852-9c77-f0c2f62cde09-kube-api-access-nwddk\") pod \"cert-manager-cainjector-559cdd8c87-vbj2p\" (UID: \"37e493ac-c476-4852-9c77-f0c2f62cde09\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vbj2p"
Jul 24 16:52:19.000981 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:19.000821    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/37e493ac-c476-4852-9c77-f0c2f62cde09-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-vbj2p\" (UID: \"37e493ac-c476-4852-9c77-f0c2f62cde09\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vbj2p"
Jul 24 16:52:19.127178 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:19.127081    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vbj2p"
Jul 24 16:52:19.253792 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:19.253609    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-vbj2p"]
Jul 24 16:52:19.256498 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:52:19.256461    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37e493ac_c476_4852_9c77_f0c2f62cde09.slice/crio-927500543e51ad99fa64769cbaebae74414953c1dfb5aadbf70205f381e4ac1b WatchSource:0}: Error finding container 927500543e51ad99fa64769cbaebae74414953c1dfb5aadbf70205f381e4ac1b: Status 404 returned error can't find the container with id 927500543e51ad99fa64769cbaebae74414953c1dfb5aadbf70205f381e4ac1b
Jul 24 16:52:20.007882 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:20.007844    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vbj2p" event={"ID":"37e493ac-c476-4852-9c77-f0c2f62cde09","Type":"ContainerStarted","Data":"1f0d1e8d12ce46c6f09c31eb6553d2ecca64a4525872cc3645f3af924acba232"}
Jul 24 16:52:20.008339 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:20.007893    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vbj2p" event={"ID":"37e493ac-c476-4852-9c77-f0c2f62cde09","Type":"ContainerStarted","Data":"927500543e51ad99fa64769cbaebae74414953c1dfb5aadbf70205f381e4ac1b"}
Jul 24 16:52:20.025692 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:20.025655    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:52:20.026509 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:20.026471    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vbj2p" podStartSLOduration=2.026456849 podStartE2EDuration="2.026456849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:52:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:52:20.026425674 +0000 UTC m=+562.668389558" watchObservedRunningTime="2026-07-24 16:52:20.026456849 +0000 UTC m=+562.668420732"
Jul 24 16:52:20.121259 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:20.121224    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:52:23.001775 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:23.001746    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-mkvf6"
Jul 24 16:52:28.166063 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:28.166025    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:52:28.260579 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:28.260548    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:52:28.498614 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:28.498514    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:52:28.553848 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:28.553815    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:52:34.274519 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:34.274480    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:52:34.375048 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:34.375015    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:52:36.469133 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:36.469097    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:52:50.714055 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:50.713995    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:52:50.761796 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:50.761767    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-675dd78886-xvb6c"]
Jul 24 16:52:50.764466 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:50.764449    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-675dd78886-xvb6c"
Jul 24 16:52:50.767023 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:50.767003    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 24 16:52:50.774084 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:50.773904    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 24 16:52:50.774187 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:50.774164    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 24 16:52:50.774254 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:50.774189    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 24 16:52:50.774316 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:50.774302    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-r7w7p\""
Jul 24 16:52:50.774431 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:50.774412    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 24 16:52:50.779764 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:50.779741    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-675dd78886-xvb6c"]
Jul 24 16:52:50.798881 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:50.798853    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:52:50.951278 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:50.951239    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/82bb361f-6db3-4b26-9d8b-117ae0929ad5-metrics-cert\") pod \"lws-controller-manager-675dd78886-xvb6c\" (UID: \"82bb361f-6db3-4b26-9d8b-117ae0929ad5\") " pod="openshift-lws-operator/lws-controller-manager-675dd78886-xvb6c"
Jul 24 16:52:50.951479 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:50.951283    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz5bq\" (UniqueName: \"kubernetes.io/projected/82bb361f-6db3-4b26-9d8b-117ae0929ad5-kube-api-access-pz5bq\") pod \"lws-controller-manager-675dd78886-xvb6c\" (UID: \"82bb361f-6db3-4b26-9d8b-117ae0929ad5\") " pod="openshift-lws-operator/lws-controller-manager-675dd78886-xvb6c"
Jul 24 16:52:50.951479 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:50.951312    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/82bb361f-6db3-4b26-9d8b-117ae0929ad5-manager-config\") pod \"lws-controller-manager-675dd78886-xvb6c\" (UID: \"82bb361f-6db3-4b26-9d8b-117ae0929ad5\") " pod="openshift-lws-operator/lws-controller-manager-675dd78886-xvb6c"
Jul 24 16:52:50.951479 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:50.951429    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/82bb361f-6db3-4b26-9d8b-117ae0929ad5-cert\") pod \"lws-controller-manager-675dd78886-xvb6c\" (UID: \"82bb361f-6db3-4b26-9d8b-117ae0929ad5\") " pod="openshift-lws-operator/lws-controller-manager-675dd78886-xvb6c"
Jul 24 16:52:51.052581 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:51.052495    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz5bq\" (UniqueName: \"kubernetes.io/projected/82bb361f-6db3-4b26-9d8b-117ae0929ad5-kube-api-access-pz5bq\") pod \"lws-controller-manager-675dd78886-xvb6c\" (UID: \"82bb361f-6db3-4b26-9d8b-117ae0929ad5\") " pod="openshift-lws-operator/lws-controller-manager-675dd78886-xvb6c"
Jul 24 16:52:51.052581 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:51.052540    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/82bb361f-6db3-4b26-9d8b-117ae0929ad5-manager-config\") pod \"lws-controller-manager-675dd78886-xvb6c\" (UID: \"82bb361f-6db3-4b26-9d8b-117ae0929ad5\") " pod="openshift-lws-operator/lws-controller-manager-675dd78886-xvb6c"
Jul 24 16:52:51.052757 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:51.052596    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/82bb361f-6db3-4b26-9d8b-117ae0929ad5-cert\") pod \"lws-controller-manager-675dd78886-xvb6c\" (UID: \"82bb361f-6db3-4b26-9d8b-117ae0929ad5\") " pod="openshift-lws-operator/lws-controller-manager-675dd78886-xvb6c"
Jul 24 16:52:51.052757 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:51.052646    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/82bb361f-6db3-4b26-9d8b-117ae0929ad5-metrics-cert\") pod \"lws-controller-manager-675dd78886-xvb6c\" (UID: \"82bb361f-6db3-4b26-9d8b-117ae0929ad5\") " pod="openshift-lws-operator/lws-controller-manager-675dd78886-xvb6c"
Jul 24 16:52:51.053254 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:51.053232    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/82bb361f-6db3-4b26-9d8b-117ae0929ad5-manager-config\") pod \"lws-controller-manager-675dd78886-xvb6c\" (UID: \"82bb361f-6db3-4b26-9d8b-117ae0929ad5\") " pod="openshift-lws-operator/lws-controller-manager-675dd78886-xvb6c"
Jul 24 16:52:51.055089 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:51.055068    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/82bb361f-6db3-4b26-9d8b-117ae0929ad5-metrics-cert\") pod \"lws-controller-manager-675dd78886-xvb6c\" (UID: \"82bb361f-6db3-4b26-9d8b-117ae0929ad5\") " pod="openshift-lws-operator/lws-controller-manager-675dd78886-xvb6c"
Jul 24 16:52:51.055177 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:51.055136    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/82bb361f-6db3-4b26-9d8b-117ae0929ad5-cert\") pod \"lws-controller-manager-675dd78886-xvb6c\" (UID: \"82bb361f-6db3-4b26-9d8b-117ae0929ad5\") " pod="openshift-lws-operator/lws-controller-manager-675dd78886-xvb6c"
Jul 24 16:52:51.065456 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:51.065431    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz5bq\" (UniqueName: \"kubernetes.io/projected/82bb361f-6db3-4b26-9d8b-117ae0929ad5-kube-api-access-pz5bq\") pod \"lws-controller-manager-675dd78886-xvb6c\" (UID: \"82bb361f-6db3-4b26-9d8b-117ae0929ad5\") " pod="openshift-lws-operator/lws-controller-manager-675dd78886-xvb6c"
Jul 24 16:52:51.081343 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:51.081314    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-675dd78886-xvb6c"
Jul 24 16:52:51.210922 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:51.210898    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-675dd78886-xvb6c"]
Jul 24 16:52:51.213404 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:52:51.213374    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82bb361f_6db3_4b26_9d8b_117ae0929ad5.slice/crio-8a87941f07de65364a8663fcac7109a446c0ca185baf9c95347d4b8c1f1ed7c3 WatchSource:0}: Error finding container 8a87941f07de65364a8663fcac7109a446c0ca185baf9c95347d4b8c1f1ed7c3: Status 404 returned error can't find the container with id 8a87941f07de65364a8663fcac7109a446c0ca185baf9c95347d4b8c1f1ed7c3
Jul 24 16:52:51.918089 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:51.918023    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:52:52.012017 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:52.011920    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:52:52.133504 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:52.133460    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-675dd78886-xvb6c" event={"ID":"82bb361f-6db3-4b26-9d8b-117ae0929ad5","Type":"ContainerStarted","Data":"8a87941f07de65364a8663fcac7109a446c0ca185baf9c95347d4b8c1f1ed7c3"}
Jul 24 16:52:52.550866 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:52.550802    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:52:52.871240 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:52.871167    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:52:52.936698 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:52.936661    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-769b58b787-964jz"]
Jul 24 16:52:52.945623 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:52.945599    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-769b58b787-964jz"
Jul 24 16:52:52.948296 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:52.948274    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 24 16:52:52.948427 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:52.948321    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 24 16:52:52.948740 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:52.948721    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 24 16:52:52.948840 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:52.948721    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-pvp4m\""
Jul 24 16:52:52.948900 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:52.948730    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 24 16:52:52.958732 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:52.958666    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-769b58b787-964jz"]
Jul 24 16:52:52.969190 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:52.969150    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:52:53.067023 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:53.066981    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:52:53.072157 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:53.072124    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6782019-c847-41fb-94bb-b6f195b3ade3-tmp\") pod \"opendatahub-operator-controller-manager-769b58b787-964jz\" (UID: \"a6782019-c847-41fb-94bb-b6f195b3ade3\") " pod="opendatahub/opendatahub-operator-controller-manager-769b58b787-964jz"
Jul 24 16:52:53.072284 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:53.072176    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a6782019-c847-41fb-94bb-b6f195b3ade3-manifests\") pod \"opendatahub-operator-controller-manager-769b58b787-964jz\" (UID: \"a6782019-c847-41fb-94bb-b6f195b3ade3\") " pod="opendatahub/opendatahub-operator-controller-manager-769b58b787-964jz"
Jul 24 16:52:53.072284 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:53.072236    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a6782019-c847-41fb-94bb-b6f195b3ade3-webhook-cert\") pod \"opendatahub-operator-controller-manager-769b58b787-964jz\" (UID: \"a6782019-c847-41fb-94bb-b6f195b3ade3\") " pod="opendatahub/opendatahub-operator-controller-manager-769b58b787-964jz"
Jul 24 16:52:53.072284 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:53.072277    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkwht\" (UniqueName: \"kubernetes.io/projected/a6782019-c847-41fb-94bb-b6f195b3ade3-kube-api-access-kkwht\") pod \"opendatahub-operator-controller-manager-769b58b787-964jz\" (UID: \"a6782019-c847-41fb-94bb-b6f195b3ade3\") " pod="opendatahub/opendatahub-operator-controller-manager-769b58b787-964jz"
Jul 24 16:52:53.072431 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:53.072298    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a6782019-c847-41fb-94bb-b6f195b3ade3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-769b58b787-964jz\" (UID: \"a6782019-c847-41fb-94bb-b6f195b3ade3\") " pod="opendatahub/opendatahub-operator-controller-manager-769b58b787-964jz"
Jul 24 16:52:53.170630 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:53.170592    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:52:53.173661 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:53.173632    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6782019-c847-41fb-94bb-b6f195b3ade3-tmp\") pod \"opendatahub-operator-controller-manager-769b58b787-964jz\" (UID: \"a6782019-c847-41fb-94bb-b6f195b3ade3\") " pod="opendatahub/opendatahub-operator-controller-manager-769b58b787-964jz"
Jul 24 16:52:53.173814 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:53.173681    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a6782019-c847-41fb-94bb-b6f195b3ade3-manifests\") pod \"opendatahub-operator-controller-manager-769b58b787-964jz\" (UID: \"a6782019-c847-41fb-94bb-b6f195b3ade3\") " pod="opendatahub/opendatahub-operator-controller-manager-769b58b787-964jz"
Jul 24 16:52:53.173814 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:53.173718    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a6782019-c847-41fb-94bb-b6f195b3ade3-webhook-cert\") pod \"opendatahub-operator-controller-manager-769b58b787-964jz\" (UID: \"a6782019-c847-41fb-94bb-b6f195b3ade3\") " pod="opendatahub/opendatahub-operator-controller-manager-769b58b787-964jz"
Jul 24 16:52:53.173814 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:53.173754    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkwht\" (UniqueName: \"kubernetes.io/projected/a6782019-c847-41fb-94bb-b6f195b3ade3-kube-api-access-kkwht\") pod \"opendatahub-operator-controller-manager-769b58b787-964jz\" (UID: \"a6782019-c847-41fb-94bb-b6f195b3ade3\") " pod="opendatahub/opendatahub-operator-controller-manager-769b58b787-964jz"
Jul 24 16:52:53.173814 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:53.173782    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a6782019-c847-41fb-94bb-b6f195b3ade3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-769b58b787-964jz\" (UID: \"a6782019-c847-41fb-94bb-b6f195b3ade3\") " pod="opendatahub/opendatahub-operator-controller-manager-769b58b787-964jz"
Jul 24 16:52:53.174162 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:53.174132    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a6782019-c847-41fb-94bb-b6f195b3ade3-manifests\") pod \"opendatahub-operator-controller-manager-769b58b787-964jz\" (UID: \"a6782019-c847-41fb-94bb-b6f195b3ade3\") " pod="opendatahub/opendatahub-operator-controller-manager-769b58b787-964jz"
Jul 24 16:52:53.176327 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:53.176288    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6782019-c847-41fb-94bb-b6f195b3ade3-tmp\") pod \"opendatahub-operator-controller-manager-769b58b787-964jz\" (UID: \"a6782019-c847-41fb-94bb-b6f195b3ade3\") " pod="opendatahub/opendatahub-operator-controller-manager-769b58b787-964jz"
Jul 24 16:52:53.176479 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:53.176456    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a6782019-c847-41fb-94bb-b6f195b3ade3-webhook-cert\") pod \"opendatahub-operator-controller-manager-769b58b787-964jz\" (UID: \"a6782019-c847-41fb-94bb-b6f195b3ade3\") " pod="opendatahub/opendatahub-operator-controller-manager-769b58b787-964jz"
Jul 24 16:52:53.176603 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:53.176587    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a6782019-c847-41fb-94bb-b6f195b3ade3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-769b58b787-964jz\" (UID: \"a6782019-c847-41fb-94bb-b6f195b3ade3\") " pod="opendatahub/opendatahub-operator-controller-manager-769b58b787-964jz"
Jul 24 16:52:53.182732 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:53.182697    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkwht\" (UniqueName: \"kubernetes.io/projected/a6782019-c847-41fb-94bb-b6f195b3ade3-kube-api-access-kkwht\") pod \"opendatahub-operator-controller-manager-769b58b787-964jz\" (UID: \"a6782019-c847-41fb-94bb-b6f195b3ade3\") " pod="opendatahub/opendatahub-operator-controller-manager-769b58b787-964jz"
Jul 24 16:52:53.259512 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:53.259473    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-769b58b787-964jz"
Jul 24 16:52:53.266851 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:53.266826    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:52:53.367861 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:53.367826    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:52:53.468506 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:53.468429    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:52:53.582522 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:53.582478    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:52:53.735064 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:53.735039    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-769b58b787-964jz"]
Jul 24 16:52:53.737356 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:52:53.737324    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6782019_c847_41fb_94bb_b6f195b3ade3.slice/crio-94033488de11c3303ec9a021e7b74bc601fa8f0542609d23a372797e7e25fa05 WatchSource:0}: Error finding container 94033488de11c3303ec9a021e7b74bc601fa8f0542609d23a372797e7e25fa05: Status 404 returned error can't find the container with id 94033488de11c3303ec9a021e7b74bc601fa8f0542609d23a372797e7e25fa05
Jul 24 16:52:54.145216 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:54.145185    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-769b58b787-964jz" event={"ID":"a6782019-c847-41fb-94bb-b6f195b3ade3","Type":"ContainerStarted","Data":"94033488de11c3303ec9a021e7b74bc601fa8f0542609d23a372797e7e25fa05"}
Jul 24 16:52:54.146908 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:54.146880    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-675dd78886-xvb6c" event={"ID":"82bb361f-6db3-4b26-9d8b-117ae0929ad5","Type":"ContainerStarted","Data":"807681c39b461a7355c4fbab4a5908679c71d5bc3ae3bcb80da082cb6582eba2"}
Jul 24 16:52:54.147052 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:54.146986    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-675dd78886-xvb6c"
Jul 24 16:52:54.168666 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:54.168622    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-675dd78886-xvb6c" podStartSLOduration=1.72812215 podStartE2EDuration="4.1686068s" totalImagesPullingTime="2.44048465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:52:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:52:54.166683797 +0000 UTC m=+596.808647684" watchObservedRunningTime="2026-07-24 16:52:54.1686068 +0000 UTC m=+596.810570685"
Jul 24 16:52:57.162604 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:57.162565    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-769b58b787-964jz" event={"ID":"a6782019-c847-41fb-94bb-b6f195b3ade3","Type":"ContainerStarted","Data":"bfc65e572cc92aa19abd6ed84d5ee6ac4ec787dd63a0abbcdd63da6e57ac46cb"}
Jul 24 16:52:57.959201 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:57.959172    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ddklx_fb5a9838-1e05-4f86-b46c-760bd7aaa038/console-operator/1.log"
Jul 24 16:52:57.960358 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:57.960337    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ddklx_fb5a9838-1e05-4f86-b46c-760bd7aaa038/console-operator/1.log"
Jul 24 16:52:58.168542 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:58.168506    2563 generic.go:356] "Generic (PLEG): container finished" podID="a6782019-c847-41fb-94bb-b6f195b3ade3" containerID="bfc65e572cc92aa19abd6ed84d5ee6ac4ec787dd63a0abbcdd63da6e57ac46cb" exitCode=0
Jul 24 16:52:58.170927 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:58.168580    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-769b58b787-964jz" event={"ID":"a6782019-c847-41fb-94bb-b6f195b3ade3","Type":"ContainerDied","Data":"bfc65e572cc92aa19abd6ed84d5ee6ac4ec787dd63a0abbcdd63da6e57ac46cb"}
Jul 24 16:52:59.175883 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:59.175847    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-769b58b787-964jz" event={"ID":"a6782019-c847-41fb-94bb-b6f195b3ade3","Type":"ContainerStarted","Data":"20105a67098a7db3ee6b67a581cb231bce4b3184affbfa6df25b9c93ffea59f7"}
Jul 24 16:52:59.176325 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:59.176025    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-769b58b787-964jz"
Jul 24 16:52:59.198198 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:52:59.198156    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-769b58b787-964jz" podStartSLOduration=2.9257869149999998 podStartE2EDuration="7.19812047s" totalImagesPullingTime="3.204124651s" totalInitContainerRuntime="1.068208904s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:52:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:52:59.196138069 +0000 UTC m=+601.838101955" watchObservedRunningTime="2026-07-24 16:52:59.19812047 +0000 UTC m=+601.840084353"
Jul 24 16:53:00.600824 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:00.600791    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:53:05.154844 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:05.154810    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-675dd78886-xvb6c"
Jul 24 16:53:10.182687 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:10.182650    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-769b58b787-964jz"
Jul 24 16:53:12.553985 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:12.553954    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:53:13.810266 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:13.810233    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:53:14.343634 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:14.343606    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:53:14.461664 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:14.461629    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:53:14.556072 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:14.556039    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:53:14.780860 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:14.780822    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:53:15.419402 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:15.419370    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:53:20.096970 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:20.096922    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:53:20.170334 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:20.170255    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:53:20.651464 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:20.651427    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-ws5gb"]
Jul 24 16:53:20.662629 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:20.662602    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ws5gb"
Jul 24 16:53:20.667289 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:20.667265    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 24 16:53:20.667598 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:20.667582    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-5sh9k\""
Jul 24 16:53:20.667710 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:20.667695    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 24 16:53:20.669929 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:20.669909    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 24 16:53:20.670057 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:20.669952    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 24 16:53:20.675404 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:20.675383    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-ws5gb"]
Jul 24 16:53:20.809435 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:20.809396    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/58b983a6-8913-4a3c-814b-2ede89bc1629-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-ws5gb\" (UID: \"58b983a6-8913-4a3c-814b-2ede89bc1629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ws5gb"
Jul 24 16:53:20.809435 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:20.809429    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/58b983a6-8913-4a3c-814b-2ede89bc1629-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-ws5gb\" (UID: \"58b983a6-8913-4a3c-814b-2ede89bc1629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ws5gb"
Jul 24 16:53:20.809676 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:20.809453    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g7jv\" (UniqueName: \"kubernetes.io/projected/58b983a6-8913-4a3c-814b-2ede89bc1629-kube-api-access-2g7jv\") pod \"istiod-openshift-gateway-5677ccb49b-ws5gb\" (UID: \"58b983a6-8913-4a3c-814b-2ede89bc1629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ws5gb"
Jul 24 16:53:20.809676 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:20.809558    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58b983a6-8913-4a3c-814b-2ede89bc1629-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-ws5gb\" (UID: \"58b983a6-8913-4a3c-814b-2ede89bc1629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ws5gb"
Jul 24 16:53:20.809676 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:20.809593    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/58b983a6-8913-4a3c-814b-2ede89bc1629-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-ws5gb\" (UID: \"58b983a6-8913-4a3c-814b-2ede89bc1629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ws5gb"
Jul 24 16:53:20.809676 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:20.809618    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/58b983a6-8913-4a3c-814b-2ede89bc1629-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-ws5gb\" (UID: \"58b983a6-8913-4a3c-814b-2ede89bc1629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ws5gb"
Jul 24 16:53:20.809851 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:20.809683    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/58b983a6-8913-4a3c-814b-2ede89bc1629-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-ws5gb\" (UID: \"58b983a6-8913-4a3c-814b-2ede89bc1629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ws5gb"
Jul 24 16:53:20.820298 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:20.820267    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:53:20.910843 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:20.910757    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/58b983a6-8913-4a3c-814b-2ede89bc1629-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-ws5gb\" (UID: \"58b983a6-8913-4a3c-814b-2ede89bc1629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ws5gb"
Jul 24 16:53:20.910843 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:20.910798    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/58b983a6-8913-4a3c-814b-2ede89bc1629-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-ws5gb\" (UID: \"58b983a6-8913-4a3c-814b-2ede89bc1629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ws5gb"
Jul 24 16:53:20.910843 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:20.910815    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/58b983a6-8913-4a3c-814b-2ede89bc1629-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-ws5gb\" (UID: \"58b983a6-8913-4a3c-814b-2ede89bc1629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ws5gb"
Jul 24 16:53:20.911105 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:20.910962    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g7jv\" (UniqueName: \"kubernetes.io/projected/58b983a6-8913-4a3c-814b-2ede89bc1629-kube-api-access-2g7jv\") pod \"istiod-openshift-gateway-5677ccb49b-ws5gb\" (UID: \"58b983a6-8913-4a3c-814b-2ede89bc1629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ws5gb"
Jul 24 16:53:20.911105 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:20.911065    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58b983a6-8913-4a3c-814b-2ede89bc1629-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-ws5gb\" (UID: \"58b983a6-8913-4a3c-814b-2ede89bc1629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ws5gb"
Jul 24 16:53:20.911105 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:20.911094    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/58b983a6-8913-4a3c-814b-2ede89bc1629-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-ws5gb\" (UID: \"58b983a6-8913-4a3c-814b-2ede89bc1629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ws5gb"
Jul 24 16:53:20.911229 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:20.911122    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/58b983a6-8913-4a3c-814b-2ede89bc1629-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-ws5gb\" (UID: \"58b983a6-8913-4a3c-814b-2ede89bc1629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ws5gb"
Jul 24 16:53:20.911867 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:20.911838    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/58b983a6-8913-4a3c-814b-2ede89bc1629-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-ws5gb\" (UID: \"58b983a6-8913-4a3c-814b-2ede89bc1629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ws5gb"
Jul 24 16:53:20.913309 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:20.913281    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/58b983a6-8913-4a3c-814b-2ede89bc1629-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-ws5gb\" (UID: \"58b983a6-8913-4a3c-814b-2ede89bc1629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ws5gb"
Jul 24 16:53:20.913309 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:20.913309    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/58b983a6-8913-4a3c-814b-2ede89bc1629-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-ws5gb\" (UID: \"58b983a6-8913-4a3c-814b-2ede89bc1629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ws5gb"
Jul 24 16:53:20.913507 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:20.913491    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/58b983a6-8913-4a3c-814b-2ede89bc1629-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-ws5gb\" (UID: \"58b983a6-8913-4a3c-814b-2ede89bc1629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ws5gb"
Jul 24 16:53:20.913670 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:20.913652    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/58b983a6-8913-4a3c-814b-2ede89bc1629-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-ws5gb\" (UID: \"58b983a6-8913-4a3c-814b-2ede89bc1629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ws5gb"
Jul 24 16:53:20.921552 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:20.921528    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58b983a6-8913-4a3c-814b-2ede89bc1629-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-ws5gb\" (UID: \"58b983a6-8913-4a3c-814b-2ede89bc1629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ws5gb"
Jul 24 16:53:20.925542 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:20.925516    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g7jv\" (UniqueName: \"kubernetes.io/projected/58b983a6-8913-4a3c-814b-2ede89bc1629-kube-api-access-2g7jv\") pod \"istiod-openshift-gateway-5677ccb49b-ws5gb\" (UID: \"58b983a6-8913-4a3c-814b-2ede89bc1629\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ws5gb"
Jul 24 16:53:20.971897 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:20.971865    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ws5gb"
Jul 24 16:53:21.131472 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:21.131440    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-ws5gb"]
Jul 24 16:53:21.134785 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:53:21.134753    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58b983a6_8913_4a3c_814b_2ede89bc1629.slice/crio-0590c274a7ccd8e4aa1a4258b813f08a05ef2041c24721232a7a81836b12b703 WatchSource:0}: Error finding container 0590c274a7ccd8e4aa1a4258b813f08a05ef2041c24721232a7a81836b12b703: Status 404 returned error can't find the container with id 0590c274a7ccd8e4aa1a4258b813f08a05ef2041c24721232a7a81836b12b703
Jul 24 16:53:21.136786 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:21.136767    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 16:53:21.261289 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:21.261211    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:53:21.279794 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:21.274865    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ws5gb" event={"ID":"58b983a6-8913-4a3c-814b-2ede89bc1629","Type":"ContainerStarted","Data":"0590c274a7ccd8e4aa1a4258b813f08a05ef2041c24721232a7a81836b12b703"}
Jul 24 16:53:21.447211 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:21.447172    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:53:23.448861 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:23.448818    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:53:23.537148 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:23.537112    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:53:23.817241 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:23.817161    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:53:23.901955 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:23.901907    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:53:24.486099 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:24.486057    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 24 16:53:24.486396 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:24.486148    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 24 16:53:25.294896 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:25.294865    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ws5gb" event={"ID":"58b983a6-8913-4a3c-814b-2ede89bc1629","Type":"ContainerStarted","Data":"7f7abef1c18f73847bbbb12cb8509f9fe810834b9d39feb368d2a2d9f533acc6"}
Jul 24 16:53:25.295090 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:25.295051    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ws5gb"
Jul 24 16:53:25.296891 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:25.296862    2563 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-ws5gb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 24 16:53:25.297093 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:25.296909    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ws5gb" podUID="58b983a6-8913-4a3c-814b-2ede89bc1629" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 16:53:25.350075 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:25.350021    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ws5gb" podStartSLOduration=2.001135732 podStartE2EDuration="5.350001765s" totalImagesPullingTime="3.348866033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:53:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:53:25.347639909 +0000 UTC m=+627.989603814" watchObservedRunningTime="2026-07-24 16:53:25.350001765 +0000 UTC m=+627.991965649"
Jul 24 16:53:26.300683 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:53:26.300653    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ws5gb"
Jul 24 16:54:14.086641 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:54:14.086605    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:54:14.181066 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:54:14.181036    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:54:14.283564 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:54:14.283528    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:54:14.861544 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:54:14.861511    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:54:14.963930 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:54:14.963898    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:54:35.746445 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:54:35.746404    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:54:35.805525 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:54:35.805490    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:54:43.142276 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:54:43.142235    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:54:46.306521 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:54:46.306484    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-848847dcf-5lw89"]
Jul 24 16:54:46.434824 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:54:46.434787    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:54:46.531021 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:54:46.530978    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:54:46.630730 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:54:46.630643    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:54:46.730681 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:54:46.730635    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:54:47.367986 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:54:47.367907    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:54:49.742106 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:54:49.742073    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:54:49.967655 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:54:49.967615    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:54:50.330121 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:54:50.330082    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:54:51.189786 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:54:51.189754    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:54:51.290547 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:54:51.290519    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:54:53.726662 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:54:53.726634    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:54:54.785428 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:54:54.785392    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:54:56.775490 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:54:56.775457    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:54:58.565920 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:54:58.565886    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:54:59.957646 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:54:59.957613    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:55:00.059305 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:00.059265    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:55:03.736787 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:03.736753    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:55:03.840736 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:03.840704    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:55:06.570224 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:06.570191    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:55:06.735532 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:06.735494    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:55:08.001058 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:08.001027    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:55:08.100815 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:08.100792    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:55:08.348498 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:08.348418    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:55:08.455002 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:08.454966    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:55:11.332403 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:11.332356    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-848847dcf-5lw89" podUID="cb0e346b-a668-47f7-b80e-982673ba98a0" containerName="console" containerID="cri-o://817f477236ff1328d97fd335940d979b75b5a8a7ccafcd986d9af189f0caba0b" gracePeriod=15
Jul 24 16:55:11.572748 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:11.572724    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-848847dcf-5lw89"
Jul 24 16:55:11.669919 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:11.669892    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cb0e346b-a668-47f7-b80e-982673ba98a0-console-config\") pod \"cb0e346b-a668-47f7-b80e-982673ba98a0\" (UID: \"cb0e346b-a668-47f7-b80e-982673ba98a0\") "
Jul 24 16:55:11.670103 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:11.669961    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cb0e346b-a668-47f7-b80e-982673ba98a0-oauth-serving-cert\") pod \"cb0e346b-a668-47f7-b80e-982673ba98a0\" (UID: \"cb0e346b-a668-47f7-b80e-982673ba98a0\") "
Jul 24 16:55:11.670103 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:11.669999    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cb0e346b-a668-47f7-b80e-982673ba98a0-console-oauth-config\") pod \"cb0e346b-a668-47f7-b80e-982673ba98a0\" (UID: \"cb0e346b-a668-47f7-b80e-982673ba98a0\") "
Jul 24 16:55:11.670103 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:11.670017    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cb0e346b-a668-47f7-b80e-982673ba98a0-service-ca\") pod \"cb0e346b-a668-47f7-b80e-982673ba98a0\" (UID: \"cb0e346b-a668-47f7-b80e-982673ba98a0\") "
Jul 24 16:55:11.670103 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:11.670035    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rnrj\" (UniqueName: \"kubernetes.io/projected/cb0e346b-a668-47f7-b80e-982673ba98a0-kube-api-access-5rnrj\") pod \"cb0e346b-a668-47f7-b80e-982673ba98a0\" (UID: \"cb0e346b-a668-47f7-b80e-982673ba98a0\") "
Jul 24 16:55:11.670103 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:11.670054    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb0e346b-a668-47f7-b80e-982673ba98a0-trusted-ca-bundle\") pod \"cb0e346b-a668-47f7-b80e-982673ba98a0\" (UID: \"cb0e346b-a668-47f7-b80e-982673ba98a0\") "
Jul 24 16:55:11.670103 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:11.670082    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb0e346b-a668-47f7-b80e-982673ba98a0-console-serving-cert\") pod \"cb0e346b-a668-47f7-b80e-982673ba98a0\" (UID: \"cb0e346b-a668-47f7-b80e-982673ba98a0\") "
Jul 24 16:55:11.670429 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:11.670277    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb0e346b-a668-47f7-b80e-982673ba98a0-console-config" (OuterVolumeSpecName: "console-config") pod "cb0e346b-a668-47f7-b80e-982673ba98a0" (UID: "cb0e346b-a668-47f7-b80e-982673ba98a0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 16:55:11.670548 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:11.670524    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb0e346b-a668-47f7-b80e-982673ba98a0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "cb0e346b-a668-47f7-b80e-982673ba98a0" (UID: "cb0e346b-a668-47f7-b80e-982673ba98a0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 16:55:11.670595 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:11.670554    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb0e346b-a668-47f7-b80e-982673ba98a0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "cb0e346b-a668-47f7-b80e-982673ba98a0" (UID: "cb0e346b-a668-47f7-b80e-982673ba98a0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 16:55:11.670634 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:11.670612    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb0e346b-a668-47f7-b80e-982673ba98a0-service-ca" (OuterVolumeSpecName: "service-ca") pod "cb0e346b-a668-47f7-b80e-982673ba98a0" (UID: "cb0e346b-a668-47f7-b80e-982673ba98a0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 16:55:11.672228 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:11.672209    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb0e346b-a668-47f7-b80e-982673ba98a0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "cb0e346b-a668-47f7-b80e-982673ba98a0" (UID: "cb0e346b-a668-47f7-b80e-982673ba98a0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 16:55:11.672404 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:11.672377    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb0e346b-a668-47f7-b80e-982673ba98a0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "cb0e346b-a668-47f7-b80e-982673ba98a0" (UID: "cb0e346b-a668-47f7-b80e-982673ba98a0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 16:55:11.672456 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:11.672438    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb0e346b-a668-47f7-b80e-982673ba98a0-kube-api-access-5rnrj" (OuterVolumeSpecName: "kube-api-access-5rnrj") pod "cb0e346b-a668-47f7-b80e-982673ba98a0" (UID: "cb0e346b-a668-47f7-b80e-982673ba98a0"). InnerVolumeSpecName "kube-api-access-5rnrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 16:55:11.765700 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:11.765669    2563 generic.go:356] "Generic (PLEG): container finished" podID="cb0e346b-a668-47f7-b80e-982673ba98a0" containerID="817f477236ff1328d97fd335940d979b75b5a8a7ccafcd986d9af189f0caba0b" exitCode=0
Jul 24 16:55:11.765849 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:11.765721    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-848847dcf-5lw89"
Jul 24 16:55:11.765849 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:11.765742    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-848847dcf-5lw89" event={"ID":"cb0e346b-a668-47f7-b80e-982673ba98a0","Type":"ContainerDied","Data":"817f477236ff1328d97fd335940d979b75b5a8a7ccafcd986d9af189f0caba0b"}
Jul 24 16:55:11.765849 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:11.765780    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-848847dcf-5lw89" event={"ID":"cb0e346b-a668-47f7-b80e-982673ba98a0","Type":"ContainerDied","Data":"db47a294cbde609c907c9d0e0308cfdf59f6ee4c5afc7db5d8f0324327da7ad6"}
Jul 24 16:55:11.765849 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:11.765801    2563 scope.go:117] "RemoveContainer" containerID="817f477236ff1328d97fd335940d979b75b5a8a7ccafcd986d9af189f0caba0b"
Jul 24 16:55:11.771095 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:11.771073    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cb0e346b-a668-47f7-b80e-982673ba98a0-console-config\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 16:55:11.771095 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:11.771095    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cb0e346b-a668-47f7-b80e-982673ba98a0-oauth-serving-cert\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 16:55:11.771264 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:11.771104    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cb0e346b-a668-47f7-b80e-982673ba98a0-console-oauth-config\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 16:55:11.771264 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:11.771113    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cb0e346b-a668-47f7-b80e-982673ba98a0-service-ca\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 16:55:11.771264 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:11.771120    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rnrj\" (UniqueName: \"kubernetes.io/projected/cb0e346b-a668-47f7-b80e-982673ba98a0-kube-api-access-5rnrj\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 16:55:11.771264 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:11.771129    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb0e346b-a668-47f7-b80e-982673ba98a0-trusted-ca-bundle\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 16:55:11.771264 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:11.771138    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb0e346b-a668-47f7-b80e-982673ba98a0-console-serving-cert\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 16:55:11.774333 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:11.774315    2563 scope.go:117] "RemoveContainer" containerID="817f477236ff1328d97fd335940d979b75b5a8a7ccafcd986d9af189f0caba0b"
Jul 24 16:55:11.774597 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:55:11.774574    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"817f477236ff1328d97fd335940d979b75b5a8a7ccafcd986d9af189f0caba0b\": container with ID starting with 817f477236ff1328d97fd335940d979b75b5a8a7ccafcd986d9af189f0caba0b not found: ID does not exist" containerID="817f477236ff1328d97fd335940d979b75b5a8a7ccafcd986d9af189f0caba0b"
Jul 24 16:55:11.774660 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:11.774608    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"817f477236ff1328d97fd335940d979b75b5a8a7ccafcd986d9af189f0caba0b"} err="failed to get container status \"817f477236ff1328d97fd335940d979b75b5a8a7ccafcd986d9af189f0caba0b\": rpc error: code = NotFound desc = could not find container \"817f477236ff1328d97fd335940d979b75b5a8a7ccafcd986d9af189f0caba0b\": container with ID starting with 817f477236ff1328d97fd335940d979b75b5a8a7ccafcd986d9af189f0caba0b not found: ID does not exist"
Jul 24 16:55:11.785717 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:11.785696    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-848847dcf-5lw89"]
Jul 24 16:55:11.789619 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:11.789598    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-848847dcf-5lw89"]
Jul 24 16:55:12.002106 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:12.002035    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb0e346b-a668-47f7-b80e-982673ba98a0" path="/var/lib/kubelet/pods/cb0e346b-a668-47f7-b80e-982673ba98a0/volumes"
Jul 24 16:55:12.145520 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:12.145487    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:55:12.245867 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:12.245835    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:55:16.881641 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:16.881606    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wbjh5"]
Jul 24 16:55:16.882044 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:16.881910    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb0e346b-a668-47f7-b80e-982673ba98a0" containerName="console"
Jul 24 16:55:16.882044 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:16.881921    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb0e346b-a668-47f7-b80e-982673ba98a0" containerName="console"
Jul 24 16:55:16.882044 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:16.882012    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb0e346b-a668-47f7-b80e-982673ba98a0" containerName="console"
Jul 24 16:55:16.885976 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:16.885955    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wbjh5"
Jul 24 16:55:16.888669 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:16.888648    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 24 16:55:16.889048 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:16.889030    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 24 16:55:16.889147 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:16.889050    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-f9j46\""
Jul 24 16:55:16.902055 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:16.902028    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wbjh5"]
Jul 24 16:55:17.013822 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:17.013788    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9fw8\" (UniqueName: \"kubernetes.io/projected/e92d9672-eb1e-4c0e-80e3-e6e920d7d933-kube-api-access-z9fw8\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-wbjh5\" (UID: \"e92d9672-eb1e-4c0e-80e3-e6e920d7d933\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wbjh5"
Jul 24 16:55:17.013994 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:17.013854    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e92d9672-eb1e-4c0e-80e3-e6e920d7d933-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-wbjh5\" (UID: \"e92d9672-eb1e-4c0e-80e3-e6e920d7d933\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wbjh5"
Jul 24 16:55:17.114842 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:17.114797    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e92d9672-eb1e-4c0e-80e3-e6e920d7d933-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-wbjh5\" (UID: \"e92d9672-eb1e-4c0e-80e3-e6e920d7d933\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wbjh5"
Jul 24 16:55:17.115030 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:17.114932    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9fw8\" (UniqueName: \"kubernetes.io/projected/e92d9672-eb1e-4c0e-80e3-e6e920d7d933-kube-api-access-z9fw8\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-wbjh5\" (UID: \"e92d9672-eb1e-4c0e-80e3-e6e920d7d933\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wbjh5"
Jul 24 16:55:17.115324 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:17.115307    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e92d9672-eb1e-4c0e-80e3-e6e920d7d933-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-wbjh5\" (UID: \"e92d9672-eb1e-4c0e-80e3-e6e920d7d933\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wbjh5"
Jul 24 16:55:17.124498 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:17.124471    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9fw8\" (UniqueName: \"kubernetes.io/projected/e92d9672-eb1e-4c0e-80e3-e6e920d7d933-kube-api-access-z9fw8\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-wbjh5\" (UID: \"e92d9672-eb1e-4c0e-80e3-e6e920d7d933\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wbjh5"
Jul 24 16:55:17.197306 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:17.197244    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wbjh5"
Jul 24 16:55:17.338208 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:17.338186    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wbjh5"]
Jul 24 16:55:17.340632 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:55:17.340608    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92d9672_eb1e_4c0e_80e3_e6e920d7d933.slice/crio-99635e32c6efa26669e1a8efb7038970b57e50bce3cf18109ad8220864eb0351 WatchSource:0}: Error finding container 99635e32c6efa26669e1a8efb7038970b57e50bce3cf18109ad8220864eb0351: Status 404 returned error can't find the container with id 99635e32c6efa26669e1a8efb7038970b57e50bce3cf18109ad8220864eb0351
Jul 24 16:55:17.794353 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:17.794317    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wbjh5" event={"ID":"e92d9672-eb1e-4c0e-80e3-e6e920d7d933","Type":"ContainerStarted","Data":"99635e32c6efa26669e1a8efb7038970b57e50bce3cf18109ad8220864eb0351"}
Jul 24 16:55:18.871885 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:18.871851    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:55:18.975298 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:18.975264    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:55:19.088606 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:19.088571    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:55:21.743170 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:21.743138    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:55:21.817954 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:21.817907    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wbjh5" event={"ID":"e92d9672-eb1e-4c0e-80e3-e6e920d7d933","Type":"ContainerStarted","Data":"d86f47a7240fac0fb2b00eddc3b5a67aa5632ca07deee482abecf523896f693b"}
Jul 24 16:55:21.818122 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:21.818058    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wbjh5"
Jul 24 16:55:21.846581 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:21.846536    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wbjh5" podStartSLOduration=1.569774655 podStartE2EDuration="5.846518848s" totalImagesPullingTime="4.276744193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:55:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:55:21.845363345 +0000 UTC m=+744.487327234" watchObservedRunningTime="2026-07-24 16:55:21.846518848 +0000 UTC m=+744.488482803"
Jul 24 16:55:21.850984 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:21.850933    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:55:26.493896 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:26.493866    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:55:26.598406 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:26.598375    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:55:30.016151 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:30.016109    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:55:30.115984 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:30.115924    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:55:32.824999 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:32.824956    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wbjh5"
Jul 24 16:55:36.652833 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:36.652796    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:55:36.754179 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:36.754148    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:55:39.951426 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:39.951346    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:55:40.039633 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:40.039601    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:55:44.454612 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:44.454571    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:55:44.555064 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:44.555029    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:55:44.658850 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:44.658800    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:55:44.713248 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:44.713138    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:55:45.341885 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:45.341853    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wbjh5"]
Jul 24 16:55:45.342072 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:45.342052    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wbjh5" podUID="e92d9672-eb1e-4c0e-80e3-e6e920d7d933" containerName="manager" containerID="cri-o://d86f47a7240fac0fb2b00eddc3b5a67aa5632ca07deee482abecf523896f693b" gracePeriod=10
Jul 24 16:55:45.592272 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:45.592217    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wbjh5"
Jul 24 16:55:45.655143 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:45.655103    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9fw8\" (UniqueName: \"kubernetes.io/projected/e92d9672-eb1e-4c0e-80e3-e6e920d7d933-kube-api-access-z9fw8\") pod \"e92d9672-eb1e-4c0e-80e3-e6e920d7d933\" (UID: \"e92d9672-eb1e-4c0e-80e3-e6e920d7d933\") "
Jul 24 16:55:45.655304 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:45.655151    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e92d9672-eb1e-4c0e-80e3-e6e920d7d933-extensions-socket-volume\") pod \"e92d9672-eb1e-4c0e-80e3-e6e920d7d933\" (UID: \"e92d9672-eb1e-4c0e-80e3-e6e920d7d933\") "
Jul 24 16:55:45.655672 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:45.655641    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e92d9672-eb1e-4c0e-80e3-e6e920d7d933-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e92d9672-eb1e-4c0e-80e3-e6e920d7d933" (UID: "e92d9672-eb1e-4c0e-80e3-e6e920d7d933"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:55:45.657337 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:45.657318    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e92d9672-eb1e-4c0e-80e3-e6e920d7d933-kube-api-access-z9fw8" (OuterVolumeSpecName: "kube-api-access-z9fw8") pod "e92d9672-eb1e-4c0e-80e3-e6e920d7d933" (UID: "e92d9672-eb1e-4c0e-80e3-e6e920d7d933"). InnerVolumeSpecName "kube-api-access-z9fw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 16:55:45.756240 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:45.756207    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9fw8\" (UniqueName: \"kubernetes.io/projected/e92d9672-eb1e-4c0e-80e3-e6e920d7d933-kube-api-access-z9fw8\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 16:55:45.756240 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:45.756233    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e92d9672-eb1e-4c0e-80e3-e6e920d7d933-extensions-socket-volume\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 16:55:45.926038 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:45.926008    2563 generic.go:356] "Generic (PLEG): container finished" podID="e92d9672-eb1e-4c0e-80e3-e6e920d7d933" containerID="d86f47a7240fac0fb2b00eddc3b5a67aa5632ca07deee482abecf523896f693b" exitCode=0
Jul 24 16:55:45.926201 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:45.926068    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wbjh5"
Jul 24 16:55:45.926201 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:45.926099    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wbjh5" event={"ID":"e92d9672-eb1e-4c0e-80e3-e6e920d7d933","Type":"ContainerDied","Data":"d86f47a7240fac0fb2b00eddc3b5a67aa5632ca07deee482abecf523896f693b"}
Jul 24 16:55:45.926201 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:45.926171    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wbjh5" event={"ID":"e92d9672-eb1e-4c0e-80e3-e6e920d7d933","Type":"ContainerDied","Data":"99635e32c6efa26669e1a8efb7038970b57e50bce3cf18109ad8220864eb0351"}
Jul 24 16:55:45.926308 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:45.926204    2563 scope.go:117] "RemoveContainer" containerID="d86f47a7240fac0fb2b00eddc3b5a67aa5632ca07deee482abecf523896f693b"
Jul 24 16:55:45.936796 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:45.936779    2563 scope.go:117] "RemoveContainer" containerID="d86f47a7240fac0fb2b00eddc3b5a67aa5632ca07deee482abecf523896f693b"
Jul 24 16:55:45.937103 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:55:45.937076    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d86f47a7240fac0fb2b00eddc3b5a67aa5632ca07deee482abecf523896f693b\": container with ID starting with d86f47a7240fac0fb2b00eddc3b5a67aa5632ca07deee482abecf523896f693b not found: ID does not exist" containerID="d86f47a7240fac0fb2b00eddc3b5a67aa5632ca07deee482abecf523896f693b"
Jul 24 16:55:45.937163 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:45.937115    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d86f47a7240fac0fb2b00eddc3b5a67aa5632ca07deee482abecf523896f693b"} err="failed to get container status \"d86f47a7240fac0fb2b00eddc3b5a67aa5632ca07deee482abecf523896f693b\": rpc error: code = NotFound desc = could not find container \"d86f47a7240fac0fb2b00eddc3b5a67aa5632ca07deee482abecf523896f693b\": container with ID starting with d86f47a7240fac0fb2b00eddc3b5a67aa5632ca07deee482abecf523896f693b not found: ID does not exist"
Jul 24 16:55:45.950822 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:45.950799    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wbjh5"]
Jul 24 16:55:45.954330 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:45.954310    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wbjh5"]
Jul 24 16:55:46.001727 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:46.001698    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e92d9672-eb1e-4c0e-80e3-e6e920d7d933" path="/var/lib/kubelet/pods/e92d9672-eb1e-4c0e-80e3-e6e920d7d933/volumes"
Jul 24 16:55:49.230844 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:49.230810    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:55:49.467113 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:49.467080    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:55:49.566188 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:49.566117    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:55:49.692290 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:49.692258    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:55:52.678131 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:55:52.678090    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:56:05.685023 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:05.684983    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-zsxp5"]
Jul 24 16:56:05.685404 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:05.685352    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e92d9672-eb1e-4c0e-80e3-e6e920d7d933" containerName="manager"
Jul 24 16:56:05.685404 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:05.685364    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e92d9672-eb1e-4c0e-80e3-e6e920d7d933" containerName="manager"
Jul 24 16:56:05.685520 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:05.685416    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e92d9672-eb1e-4c0e-80e3-e6e920d7d933" containerName="manager"
Jul 24 16:56:05.688678 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:05.688638    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:56:05.689746 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:05.689730    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-zsxp5"
Jul 24 16:56:05.692376 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:05.692352    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 24 16:56:05.692512 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:05.692352    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 24 16:56:05.693327 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:05.693305    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-dwnsj\""
Jul 24 16:56:05.693380 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:05.693315    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 24 16:56:05.697980 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:05.697959    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-zsxp5"]
Jul 24 16:56:05.722866 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:05.722830    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmch9\" (UniqueName: \"kubernetes.io/projected/1f11f0c3-1e98-43df-ab17-1eb905d93664-kube-api-access-nmch9\") pod \"limitador-limitador-64c94fd64f-zsxp5\" (UID: \"1f11f0c3-1e98-43df-ab17-1eb905d93664\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-zsxp5"
Jul 24 16:56:05.723062 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:05.722881    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1f11f0c3-1e98-43df-ab17-1eb905d93664-config-file\") pod \"limitador-limitador-64c94fd64f-zsxp5\" (UID: \"1f11f0c3-1e98-43df-ab17-1eb905d93664\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-zsxp5"
Jul 24 16:56:05.777630 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:05.777585    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-zsxp5"]
Jul 24 16:56:05.786888 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:05.786837    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:56:05.823741 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:05.823707    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmch9\" (UniqueName: \"kubernetes.io/projected/1f11f0c3-1e98-43df-ab17-1eb905d93664-kube-api-access-nmch9\") pod \"limitador-limitador-64c94fd64f-zsxp5\" (UID: \"1f11f0c3-1e98-43df-ab17-1eb905d93664\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-zsxp5"
Jul 24 16:56:05.823933 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:05.823757    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1f11f0c3-1e98-43df-ab17-1eb905d93664-config-file\") pod \"limitador-limitador-64c94fd64f-zsxp5\" (UID: \"1f11f0c3-1e98-43df-ab17-1eb905d93664\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-zsxp5"
Jul 24 16:56:05.824408 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:05.824390    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1f11f0c3-1e98-43df-ab17-1eb905d93664-config-file\") pod \"limitador-limitador-64c94fd64f-zsxp5\" (UID: \"1f11f0c3-1e98-43df-ab17-1eb905d93664\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-zsxp5"
Jul 24 16:56:05.832222 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:05.832198    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmch9\" (UniqueName: \"kubernetes.io/projected/1f11f0c3-1e98-43df-ab17-1eb905d93664-kube-api-access-nmch9\") pod \"limitador-limitador-64c94fd64f-zsxp5\" (UID: \"1f11f0c3-1e98-43df-ab17-1eb905d93664\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-zsxp5"
Jul 24 16:56:05.844591 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:05.844563    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:56:06.000417 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:06.000338    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-zsxp5"
Jul 24 16:56:06.130570 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:06.130537    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-zsxp5"]
Jul 24 16:56:06.133658 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:56:06.133626    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f11f0c3_1e98_43df_ab17_1eb905d93664.slice/crio-97aed546df465ecbed77b24b8c551602d989554ccaf21c3405156032a5605332 WatchSource:0}: Error finding container 97aed546df465ecbed77b24b8c551602d989554ccaf21c3405156032a5605332: Status 404 returned error can't find the container with id 97aed546df465ecbed77b24b8c551602d989554ccaf21c3405156032a5605332
Jul 24 16:56:06.516518 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:06.516482    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:56:06.617325 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:06.617295    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:56:07.025288 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:07.025249    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-zsxp5" event={"ID":"1f11f0c3-1e98-43df-ab17-1eb905d93664","Type":"ContainerStarted","Data":"97aed546df465ecbed77b24b8c551602d989554ccaf21c3405156032a5605332"}
Jul 24 16:56:09.035733 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:09.035701    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-zsxp5" event={"ID":"1f11f0c3-1e98-43df-ab17-1eb905d93664","Type":"ContainerStarted","Data":"e1b354d5142603fcab27616fc5e16ac73c6361bacb6f12c99520ed6dafbc7f68"}
Jul 24 16:56:09.036148 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:09.035828    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-zsxp5"
Jul 24 16:56:09.053443 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:09.053400    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-zsxp5" podStartSLOduration=1.317128958 podStartE2EDuration="4.053387854s" totalImagesPullingTime="2.736258896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:56:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:56:09.052167807 +0000 UTC m=+791.694131691" watchObservedRunningTime="2026-07-24 16:56:09.053387854 +0000 UTC m=+791.695351737"
Jul 24 16:56:20.042165 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:20.042136    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-zsxp5"
Jul 24 16:56:20.798144 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:20.798102    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-zsxp5"]
Jul 24 16:56:20.798410 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:20.798362    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-zsxp5" podUID="1f11f0c3-1e98-43df-ab17-1eb905d93664" containerName="limitador" containerID="cri-o://e1b354d5142603fcab27616fc5e16ac73c6361bacb6f12c99520ed6dafbc7f68" gracePeriod=30
Jul 24 16:56:21.347178 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:21.347152    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-zsxp5"
Jul 24 16:56:21.456010 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:21.455975    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmch9\" (UniqueName: \"kubernetes.io/projected/1f11f0c3-1e98-43df-ab17-1eb905d93664-kube-api-access-nmch9\") pod \"1f11f0c3-1e98-43df-ab17-1eb905d93664\" (UID: \"1f11f0c3-1e98-43df-ab17-1eb905d93664\") "
Jul 24 16:56:21.456177 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:21.456053    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1f11f0c3-1e98-43df-ab17-1eb905d93664-config-file\") pod \"1f11f0c3-1e98-43df-ab17-1eb905d93664\" (UID: \"1f11f0c3-1e98-43df-ab17-1eb905d93664\") "
Jul 24 16:56:21.456406 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:21.456378    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f11f0c3-1e98-43df-ab17-1eb905d93664-config-file" (OuterVolumeSpecName: "config-file") pod "1f11f0c3-1e98-43df-ab17-1eb905d93664" (UID: "1f11f0c3-1e98-43df-ab17-1eb905d93664"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 16:56:21.458084 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:21.458058    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f11f0c3-1e98-43df-ab17-1eb905d93664-kube-api-access-nmch9" (OuterVolumeSpecName: "kube-api-access-nmch9") pod "1f11f0c3-1e98-43df-ab17-1eb905d93664" (UID: "1f11f0c3-1e98-43df-ab17-1eb905d93664"). InnerVolumeSpecName "kube-api-access-nmch9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 16:56:21.556723 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:21.556688    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmch9\" (UniqueName: \"kubernetes.io/projected/1f11f0c3-1e98-43df-ab17-1eb905d93664-kube-api-access-nmch9\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 16:56:21.556723 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:21.556715    2563 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1f11f0c3-1e98-43df-ab17-1eb905d93664-config-file\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 16:56:22.090980 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:22.090929    2563 generic.go:356] "Generic (PLEG): container finished" podID="1f11f0c3-1e98-43df-ab17-1eb905d93664" containerID="e1b354d5142603fcab27616fc5e16ac73c6361bacb6f12c99520ed6dafbc7f68" exitCode=0
Jul 24 16:56:22.091154 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:22.091002    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-zsxp5"
Jul 24 16:56:22.091154 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:22.091035    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-zsxp5" event={"ID":"1f11f0c3-1e98-43df-ab17-1eb905d93664","Type":"ContainerDied","Data":"e1b354d5142603fcab27616fc5e16ac73c6361bacb6f12c99520ed6dafbc7f68"}
Jul 24 16:56:22.091154 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:22.091087    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-zsxp5" event={"ID":"1f11f0c3-1e98-43df-ab17-1eb905d93664","Type":"ContainerDied","Data":"97aed546df465ecbed77b24b8c551602d989554ccaf21c3405156032a5605332"}
Jul 24 16:56:22.091154 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:22.091109    2563 scope.go:117] "RemoveContainer" containerID="e1b354d5142603fcab27616fc5e16ac73c6361bacb6f12c99520ed6dafbc7f68"
Jul 24 16:56:22.099229 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:22.099213    2563 scope.go:117] "RemoveContainer" containerID="e1b354d5142603fcab27616fc5e16ac73c6361bacb6f12c99520ed6dafbc7f68"
Jul 24 16:56:22.099479 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:56:22.099461    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1b354d5142603fcab27616fc5e16ac73c6361bacb6f12c99520ed6dafbc7f68\": container with ID starting with e1b354d5142603fcab27616fc5e16ac73c6361bacb6f12c99520ed6dafbc7f68 not found: ID does not exist" containerID="e1b354d5142603fcab27616fc5e16ac73c6361bacb6f12c99520ed6dafbc7f68"
Jul 24 16:56:22.099532 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:22.099488    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1b354d5142603fcab27616fc5e16ac73c6361bacb6f12c99520ed6dafbc7f68"} err="failed to get container status \"e1b354d5142603fcab27616fc5e16ac73c6361bacb6f12c99520ed6dafbc7f68\": rpc error: code = NotFound desc = could not find container \"e1b354d5142603fcab27616fc5e16ac73c6361bacb6f12c99520ed6dafbc7f68\": container with ID starting with e1b354d5142603fcab27616fc5e16ac73c6361bacb6f12c99520ed6dafbc7f68 not found: ID does not exist"
Jul 24 16:56:22.107951 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:22.107918    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-zsxp5"]
Jul 24 16:56:22.113749 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:22.113725    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-zsxp5"]
Jul 24 16:56:22.887667 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:22.887612    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:56:22.937925 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:22.937867    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:56:23.126877 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:23.126841    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:56:23.226856 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:23.226757    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:56:23.564597 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:23.564524    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-5qhkt"]
Jul 24 16:56:23.564880 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:23.564866    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f11f0c3-1e98-43df-ab17-1eb905d93664" containerName="limitador"
Jul 24 16:56:23.564880 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:23.564880    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f11f0c3-1e98-43df-ab17-1eb905d93664" containerName="limitador"
Jul 24 16:56:23.565052 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:23.564954    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f11f0c3-1e98-43df-ab17-1eb905d93664" containerName="limitador"
Jul 24 16:56:23.567980 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:23.567959    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5qhkt"
Jul 24 16:56:23.571570 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:23.571538    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 24 16:56:23.571685 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:23.571579    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 16:56:23.571685 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:23.571579    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-45bkm\""
Jul 24 16:56:23.571685 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:23.571579    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 16:56:23.576062 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:23.576042    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-5qhkt"]
Jul 24 16:56:23.599826 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:23.599798    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:56:23.676917 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:23.676885    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpzbj\" (UniqueName: \"kubernetes.io/projected/b017af45-b099-4a44-877d-ce1b1530c07b-kube-api-access-fpzbj\") pod \"postgres-69fb4594cf-5qhkt\" (UID: \"b017af45-b099-4a44-877d-ce1b1530c07b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5qhkt"
Jul 24 16:56:23.677082 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:23.676983    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b017af45-b099-4a44-877d-ce1b1530c07b-data\") pod \"postgres-69fb4594cf-5qhkt\" (UID: \"b017af45-b099-4a44-877d-ce1b1530c07b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5qhkt"
Jul 24 16:56:23.778464 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:23.778410    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpzbj\" (UniqueName: \"kubernetes.io/projected/b017af45-b099-4a44-877d-ce1b1530c07b-kube-api-access-fpzbj\") pod \"postgres-69fb4594cf-5qhkt\" (UID: \"b017af45-b099-4a44-877d-ce1b1530c07b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5qhkt"
Jul 24 16:56:23.778663 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:23.778562    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b017af45-b099-4a44-877d-ce1b1530c07b-data\") pod \"postgres-69fb4594cf-5qhkt\" (UID: \"b017af45-b099-4a44-877d-ce1b1530c07b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5qhkt"
Jul 24 16:56:23.779000 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:23.778978    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b017af45-b099-4a44-877d-ce1b1530c07b-data\") pod \"postgres-69fb4594cf-5qhkt\" (UID: \"b017af45-b099-4a44-877d-ce1b1530c07b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5qhkt"
Jul 24 16:56:23.786419 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:23.786401    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpzbj\" (UniqueName: \"kubernetes.io/projected/b017af45-b099-4a44-877d-ce1b1530c07b-kube-api-access-fpzbj\") pod \"postgres-69fb4594cf-5qhkt\" (UID: \"b017af45-b099-4a44-877d-ce1b1530c07b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5qhkt"
Jul 24 16:56:23.879917 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:23.879826    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5qhkt"
Jul 24 16:56:24.002356 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:24.002325    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f11f0c3-1e98-43df-ab17-1eb905d93664" path="/var/lib/kubelet/pods/1f11f0c3-1e98-43df-ab17-1eb905d93664/volumes"
Jul 24 16:56:24.009259 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:24.009239    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-5qhkt"]
Jul 24 16:56:24.011760 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:56:24.011734    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb017af45_b099_4a44_877d_ce1b1530c07b.slice/crio-c06b67e2c037cf57e3c036ec7f6ea5b40e5b5300d3cf919351b4aea0b08d07bb WatchSource:0}: Error finding container c06b67e2c037cf57e3c036ec7f6ea5b40e5b5300d3cf919351b4aea0b08d07bb: Status 404 returned error can't find the container with id c06b67e2c037cf57e3c036ec7f6ea5b40e5b5300d3cf919351b4aea0b08d07bb
Jul 24 16:56:24.102852 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:24.102820    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5qhkt" event={"ID":"b017af45-b099-4a44-877d-ce1b1530c07b","Type":"ContainerStarted","Data":"c06b67e2c037cf57e3c036ec7f6ea5b40e5b5300d3cf919351b4aea0b08d07bb"}
Jul 24 16:56:29.131663 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:29.131636    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5qhkt" event={"ID":"b017af45-b099-4a44-877d-ce1b1530c07b","Type":"ContainerStarted","Data":"5eccab23d0def61951d1670c500955e4689d12812f841a29ebb2cc215e2f6324"}
Jul 24 16:56:29.132013 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:29.131714    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5qhkt"
Jul 24 16:56:29.149310 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:29.149261    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5qhkt" podStartSLOduration=1.371982274 podStartE2EDuration="6.149246877s" totalImagesPullingTime="4.777264603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:56:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:56:29.146757217 +0000 UTC m=+811.788721101" watchObservedRunningTime="2026-07-24 16:56:29.149246877 +0000 UTC m=+811.791210781"
Jul 24 16:56:35.164002 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:35.163974    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5qhkt"
Jul 24 16:56:35.495712 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:35.495632    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:56:35.716335 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:35.716302    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:56:42.311000 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:42.310965    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:56:42.318479 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:42.318447    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84f674679d-s5sp5"]
Jul 24 16:56:42.349048 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:42.349010    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84f674679d-s5sp5"]
Jul 24 16:56:42.349218 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:42.349126    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84f674679d-s5sp5"
Jul 24 16:56:42.351699 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:42.351673    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 24 16:56:42.352426 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:42.352152    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-6x8jt\""
Jul 24 16:56:42.352426 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:42.352151    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 24 16:56:42.395898 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:42.395869    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:56:42.434052 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:42.434024    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mttd\" (UniqueName: \"kubernetes.io/projected/f3b88d5d-0dac-4175-b820-fd9be768a7f3-kube-api-access-5mttd\") pod \"maas-controller-84f674679d-s5sp5\" (UID: \"f3b88d5d-0dac-4175-b820-fd9be768a7f3\") " pod="opendatahub/maas-controller-84f674679d-s5sp5"
Jul 24 16:56:42.434169 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:42.434077    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f3b88d5d-0dac-4175-b820-fd9be768a7f3-webhook-cert\") pod \"maas-controller-84f674679d-s5sp5\" (UID: \"f3b88d5d-0dac-4175-b820-fd9be768a7f3\") " pod="opendatahub/maas-controller-84f674679d-s5sp5"
Jul 24 16:56:42.449630 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:42.449605    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:56:42.510844 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:42.510818    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7d8c754f6b-npwxj"]
Jul 24 16:56:42.514305 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:42.514291    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d8c754f6b-npwxj"
Jul 24 16:56:42.523172 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:42.523147    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d8c754f6b-npwxj"]
Jul 24 16:56:42.529487 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:42.529464    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:56:42.534734 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:42.534714    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mttd\" (UniqueName: \"kubernetes.io/projected/f3b88d5d-0dac-4175-b820-fd9be768a7f3-kube-api-access-5mttd\") pod \"maas-controller-84f674679d-s5sp5\" (UID: \"f3b88d5d-0dac-4175-b820-fd9be768a7f3\") " pod="opendatahub/maas-controller-84f674679d-s5sp5"
Jul 24 16:56:42.534809 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:42.534757    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f3b88d5d-0dac-4175-b820-fd9be768a7f3-webhook-cert\") pod \"maas-controller-84f674679d-s5sp5\" (UID: \"f3b88d5d-0dac-4175-b820-fd9be768a7f3\") " pod="opendatahub/maas-controller-84f674679d-s5sp5"
Jul 24 16:56:42.537331 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:42.537292    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f3b88d5d-0dac-4175-b820-fd9be768a7f3-webhook-cert\") pod \"maas-controller-84f674679d-s5sp5\" (UID: \"f3b88d5d-0dac-4175-b820-fd9be768a7f3\") " pod="opendatahub/maas-controller-84f674679d-s5sp5"
Jul 24 16:56:42.543076 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:42.543054    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mttd\" (UniqueName: \"kubernetes.io/projected/f3b88d5d-0dac-4175-b820-fd9be768a7f3-kube-api-access-5mttd\") pod \"maas-controller-84f674679d-s5sp5\" (UID: \"f3b88d5d-0dac-4175-b820-fd9be768a7f3\") " pod="opendatahub/maas-controller-84f674679d-s5sp5"
Jul 24 16:56:42.635795 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:42.635764    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b56649dc-ecfc-4258-885b-484147067106-webhook-cert\") pod \"maas-controller-7d8c754f6b-npwxj\" (UID: \"b56649dc-ecfc-4258-885b-484147067106\") " pod="opendatahub/maas-controller-7d8c754f6b-npwxj"
Jul 24 16:56:42.635981 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:42.635829    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkfng\" (UniqueName: \"kubernetes.io/projected/b56649dc-ecfc-4258-885b-484147067106-kube-api-access-zkfng\") pod \"maas-controller-7d8c754f6b-npwxj\" (UID: \"b56649dc-ecfc-4258-885b-484147067106\") " pod="opendatahub/maas-controller-7d8c754f6b-npwxj"
Jul 24 16:56:42.663263 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:42.663242    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84f674679d-s5sp5"
Jul 24 16:56:42.736951 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:42.736906    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkfng\" (UniqueName: \"kubernetes.io/projected/b56649dc-ecfc-4258-885b-484147067106-kube-api-access-zkfng\") pod \"maas-controller-7d8c754f6b-npwxj\" (UID: \"b56649dc-ecfc-4258-885b-484147067106\") " pod="opendatahub/maas-controller-7d8c754f6b-npwxj"
Jul 24 16:56:42.737110 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:42.737022    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b56649dc-ecfc-4258-885b-484147067106-webhook-cert\") pod \"maas-controller-7d8c754f6b-npwxj\" (UID: \"b56649dc-ecfc-4258-885b-484147067106\") " pod="opendatahub/maas-controller-7d8c754f6b-npwxj"
Jul 24 16:56:42.739324 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:42.739299    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b56649dc-ecfc-4258-885b-484147067106-webhook-cert\") pod \"maas-controller-7d8c754f6b-npwxj\" (UID: \"b56649dc-ecfc-4258-885b-484147067106\") " pod="opendatahub/maas-controller-7d8c754f6b-npwxj"
Jul 24 16:56:42.746400 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:42.746338    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkfng\" (UniqueName: \"kubernetes.io/projected/b56649dc-ecfc-4258-885b-484147067106-kube-api-access-zkfng\") pod \"maas-controller-7d8c754f6b-npwxj\" (UID: \"b56649dc-ecfc-4258-885b-484147067106\") " pod="opendatahub/maas-controller-7d8c754f6b-npwxj"
Jul 24 16:56:42.792421 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:42.792288    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84f674679d-s5sp5"]
Jul 24 16:56:42.795213 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:56:42.795189    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3b88d5d_0dac_4175_b820_fd9be768a7f3.slice/crio-31c65f626e71eddbc4702824da9fdc8d453b66fbd3661747f584366e4e7f885d WatchSource:0}: Error finding container 31c65f626e71eddbc4702824da9fdc8d453b66fbd3661747f584366e4e7f885d: Status 404 returned error can't find the container with id 31c65f626e71eddbc4702824da9fdc8d453b66fbd3661747f584366e4e7f885d
Jul 24 16:56:42.825299 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:42.825270    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d8c754f6b-npwxj"
Jul 24 16:56:42.951490 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:42.951371    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d8c754f6b-npwxj"]
Jul 24 16:56:42.953583 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:56:42.953558    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb56649dc_ecfc_4258_885b_484147067106.slice/crio-3b6ee874f0fb829a94a428c90dd167f88bed0372a10aa6fe7d2c323b85e50d8a WatchSource:0}: Error finding container 3b6ee874f0fb829a94a428c90dd167f88bed0372a10aa6fe7d2c323b85e50d8a: Status 404 returned error can't find the container with id 3b6ee874f0fb829a94a428c90dd167f88bed0372a10aa6fe7d2c323b85e50d8a
Jul 24 16:56:43.191354 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:43.191261    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d8c754f6b-npwxj" event={"ID":"b56649dc-ecfc-4258-885b-484147067106","Type":"ContainerStarted","Data":"3b6ee874f0fb829a94a428c90dd167f88bed0372a10aa6fe7d2c323b85e50d8a"}
Jul 24 16:56:43.192353 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:43.192330    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84f674679d-s5sp5" event={"ID":"f3b88d5d-0dac-4175-b820-fd9be768a7f3","Type":"ContainerStarted","Data":"31c65f626e71eddbc4702824da9fdc8d453b66fbd3661747f584366e4e7f885d"}
Jul 24 16:56:45.578138 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:45.578106    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:56:45.675310 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:45.675231    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:56:46.210202 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:46.210169    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84f674679d-s5sp5" event={"ID":"f3b88d5d-0dac-4175-b820-fd9be768a7f3","Type":"ContainerStarted","Data":"361431a2023bd63ddb687a11902f64d4d4a4134639dcff32a07113ea5fb3916d"}
Jul 24 16:56:46.210560 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:46.210257    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84f674679d-s5sp5"
Jul 24 16:56:46.211616 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:46.211596    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d8c754f6b-npwxj" event={"ID":"b56649dc-ecfc-4258-885b-484147067106","Type":"ContainerStarted","Data":"875c9c33551da006a3bab55aaa1d34fe15001143c85125530d898c0ce357af19"}
Jul 24 16:56:46.211731 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:46.211717    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7d8c754f6b-npwxj"
Jul 24 16:56:46.239732 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:46.239675    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-84f674679d-s5sp5" podStartSLOduration=1.633914514 podStartE2EDuration="4.239659406s" totalImagesPullingTime="2.605744892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:56:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:56:46.238176467 +0000 UTC m=+828.880140370" watchObservedRunningTime="2026-07-24 16:56:46.239659406 +0000 UTC m=+828.881623341"
Jul 24 16:56:46.272808 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:46.272758    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7d8c754f6b-npwxj" podStartSLOduration=1.815005186 podStartE2EDuration="4.27274298s" totalImagesPullingTime="2.457737794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:56:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:56:46.272543701 +0000 UTC m=+828.914507621" watchObservedRunningTime="2026-07-24 16:56:46.27274298 +0000 UTC m=+828.914706863"
Jul 24 16:56:57.217848 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:57.217818    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84f674679d-s5sp5"
Jul 24 16:56:57.218210 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:57.217882    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7d8c754f6b-npwxj"
Jul 24 16:56:57.277530 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:57.277501    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84f674679d-s5sp5"]
Jul 24 16:56:57.277784 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:57.277756    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-84f674679d-s5sp5" podUID="f3b88d5d-0dac-4175-b820-fd9be768a7f3" containerName="manager" containerID="cri-o://361431a2023bd63ddb687a11902f64d4d4a4134639dcff32a07113ea5fb3916d" gracePeriod=30
Jul 24 16:56:57.524085 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:57.524061    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84f674679d-s5sp5"
Jul 24 16:56:57.569138 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:57.569114    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f3b88d5d-0dac-4175-b820-fd9be768a7f3-webhook-cert\") pod \"f3b88d5d-0dac-4175-b820-fd9be768a7f3\" (UID: \"f3b88d5d-0dac-4175-b820-fd9be768a7f3\") "
Jul 24 16:56:57.569258 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:57.569203    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mttd\" (UniqueName: \"kubernetes.io/projected/f3b88d5d-0dac-4175-b820-fd9be768a7f3-kube-api-access-5mttd\") pod \"f3b88d5d-0dac-4175-b820-fd9be768a7f3\" (UID: \"f3b88d5d-0dac-4175-b820-fd9be768a7f3\") "
Jul 24 16:56:57.571303 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:57.571275    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3b88d5d-0dac-4175-b820-fd9be768a7f3-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f3b88d5d-0dac-4175-b820-fd9be768a7f3" (UID: "f3b88d5d-0dac-4175-b820-fd9be768a7f3"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 16:56:57.571411 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:57.571332    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b88d5d-0dac-4175-b820-fd9be768a7f3-kube-api-access-5mttd" (OuterVolumeSpecName: "kube-api-access-5mttd") pod "f3b88d5d-0dac-4175-b820-fd9be768a7f3" (UID: "f3b88d5d-0dac-4175-b820-fd9be768a7f3"). InnerVolumeSpecName "kube-api-access-5mttd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 16:56:57.670265 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:57.670244    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mttd\" (UniqueName: \"kubernetes.io/projected/f3b88d5d-0dac-4175-b820-fd9be768a7f3-kube-api-access-5mttd\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 16:56:57.670265 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:57.670264    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f3b88d5d-0dac-4175-b820-fd9be768a7f3-webhook-cert\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 16:56:58.266732 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:58.266696    2563 generic.go:356] "Generic (PLEG): container finished" podID="f3b88d5d-0dac-4175-b820-fd9be768a7f3" containerID="361431a2023bd63ddb687a11902f64d4d4a4134639dcff32a07113ea5fb3916d" exitCode=0
Jul 24 16:56:58.266732 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:58.266736    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84f674679d-s5sp5" event={"ID":"f3b88d5d-0dac-4175-b820-fd9be768a7f3","Type":"ContainerDied","Data":"361431a2023bd63ddb687a11902f64d4d4a4134639dcff32a07113ea5fb3916d"}
Jul 24 16:56:58.267153 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:58.266755    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84f674679d-s5sp5"
Jul 24 16:56:58.267153 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:58.266800    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84f674679d-s5sp5" event={"ID":"f3b88d5d-0dac-4175-b820-fd9be768a7f3","Type":"ContainerDied","Data":"31c65f626e71eddbc4702824da9fdc8d453b66fbd3661747f584366e4e7f885d"}
Jul 24 16:56:58.267153 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:58.266820    2563 scope.go:117] "RemoveContainer" containerID="361431a2023bd63ddb687a11902f64d4d4a4134639dcff32a07113ea5fb3916d"
Jul 24 16:56:58.274640 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:58.274612    2563 scope.go:117] "RemoveContainer" containerID="361431a2023bd63ddb687a11902f64d4d4a4134639dcff32a07113ea5fb3916d"
Jul 24 16:56:58.274892 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:56:58.274869    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"361431a2023bd63ddb687a11902f64d4d4a4134639dcff32a07113ea5fb3916d\": container with ID starting with 361431a2023bd63ddb687a11902f64d4d4a4134639dcff32a07113ea5fb3916d not found: ID does not exist" containerID="361431a2023bd63ddb687a11902f64d4d4a4134639dcff32a07113ea5fb3916d"
Jul 24 16:56:58.274967 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:58.274903    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"361431a2023bd63ddb687a11902f64d4d4a4134639dcff32a07113ea5fb3916d"} err="failed to get container status \"361431a2023bd63ddb687a11902f64d4d4a4134639dcff32a07113ea5fb3916d\": rpc error: code = NotFound desc = could not find container \"361431a2023bd63ddb687a11902f64d4d4a4134639dcff32a07113ea5fb3916d\": container with ID starting with 361431a2023bd63ddb687a11902f64d4d4a4134639dcff32a07113ea5fb3916d not found: ID does not exist"
Jul 24 16:56:58.283007 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:58.282985    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84f674679d-s5sp5"]
Jul 24 16:56:58.287214 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:56:58.287194    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-84f674679d-s5sp5"]
Jul 24 16:57:00.002337 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:00.002296    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3b88d5d-0dac-4175-b820-fd9be768a7f3" path="/var/lib/kubelet/pods/f3b88d5d-0dac-4175-b820-fd9be768a7f3/volumes"
Jul 24 16:57:17.263480 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:17.263434    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:57:17.363685 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:17.363649    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:57:17.885669 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:17.885636    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:57:17.971189 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:17.971154    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-67c5c4478-779b7"]
Jul 24 16:57:17.971551 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:17.971536    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3b88d5d-0dac-4175-b820-fd9be768a7f3" containerName="manager"
Jul 24 16:57:17.971595 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:17.971553    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3b88d5d-0dac-4175-b820-fd9be768a7f3" containerName="manager"
Jul 24 16:57:17.971640 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:17.971630    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3b88d5d-0dac-4175-b820-fd9be768a7f3" containerName="manager"
Jul 24 16:57:17.976102 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:17.976083    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67c5c4478-779b7"
Jul 24 16:57:17.979280 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:17.979260    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hqnrd\""
Jul 24 16:57:17.979542 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:17.979526    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 24 16:57:17.987568 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:17.987547    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-67c5c4478-779b7"]
Jul 24 16:57:18.032395 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:18.032362    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/497455f7-9ae1-499a-be43-e74bb3cc5fc3-openshift-service-ca\") pod \"maas-api-67c5c4478-779b7\" (UID: \"497455f7-9ae1-499a-be43-e74bb3cc5fc3\") " pod="odh-ai-gateway-infra/maas-api-67c5c4478-779b7"
Jul 24 16:57:18.032576 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:18.032420    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzbm8\" (UniqueName: \"kubernetes.io/projected/497455f7-9ae1-499a-be43-e74bb3cc5fc3-kube-api-access-hzbm8\") pod \"maas-api-67c5c4478-779b7\" (UID: \"497455f7-9ae1-499a-be43-e74bb3cc5fc3\") " pod="odh-ai-gateway-infra/maas-api-67c5c4478-779b7"
Jul 24 16:57:18.032636 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:18.032605    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/497455f7-9ae1-499a-be43-e74bb3cc5fc3-maas-api-tls\") pod \"maas-api-67c5c4478-779b7\" (UID: \"497455f7-9ae1-499a-be43-e74bb3cc5fc3\") " pod="odh-ai-gateway-infra/maas-api-67c5c4478-779b7"
Jul 24 16:57:18.133356 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:18.133316    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/497455f7-9ae1-499a-be43-e74bb3cc5fc3-maas-api-tls\") pod \"maas-api-67c5c4478-779b7\" (UID: \"497455f7-9ae1-499a-be43-e74bb3cc5fc3\") " pod="odh-ai-gateway-infra/maas-api-67c5c4478-779b7"
Jul 24 16:57:18.133548 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:18.133377    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/497455f7-9ae1-499a-be43-e74bb3cc5fc3-openshift-service-ca\") pod \"maas-api-67c5c4478-779b7\" (UID: \"497455f7-9ae1-499a-be43-e74bb3cc5fc3\") " pod="odh-ai-gateway-infra/maas-api-67c5c4478-779b7"
Jul 24 16:57:18.133548 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:18.133410    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzbm8\" (UniqueName: \"kubernetes.io/projected/497455f7-9ae1-499a-be43-e74bb3cc5fc3-kube-api-access-hzbm8\") pod \"maas-api-67c5c4478-779b7\" (UID: \"497455f7-9ae1-499a-be43-e74bb3cc5fc3\") " pod="odh-ai-gateway-infra/maas-api-67c5c4478-779b7"
Jul 24 16:57:18.133548 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:57:18.133469    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Jul 24 16:57:18.133548 ip-10-0-133-234 kubenswrapper[2563]: E0724 16:57:18.133538    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/497455f7-9ae1-499a-be43-e74bb3cc5fc3-maas-api-tls podName:497455f7-9ae1-499a-be43-e74bb3cc5fc3 nodeName:}" failed. No retries permitted until 2026-07-24 16:57:18.633517228 +0000 UTC m=+861.275481106 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/497455f7-9ae1-499a-be43-e74bb3cc5fc3-maas-api-tls") pod "maas-api-67c5c4478-779b7" (UID: "497455f7-9ae1-499a-be43-e74bb3cc5fc3") : secret "maas-api-serving-cert" not found
Jul 24 16:57:18.134082 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:18.134054    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/497455f7-9ae1-499a-be43-e74bb3cc5fc3-openshift-service-ca\") pod \"maas-api-67c5c4478-779b7\" (UID: \"497455f7-9ae1-499a-be43-e74bb3cc5fc3\") " pod="odh-ai-gateway-infra/maas-api-67c5c4478-779b7"
Jul 24 16:57:18.149346 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:18.149291    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzbm8\" (UniqueName: \"kubernetes.io/projected/497455f7-9ae1-499a-be43-e74bb3cc5fc3-kube-api-access-hzbm8\") pod \"maas-api-67c5c4478-779b7\" (UID: \"497455f7-9ae1-499a-be43-e74bb3cc5fc3\") " pod="odh-ai-gateway-infra/maas-api-67c5c4478-779b7"
Jul 24 16:57:18.317901 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:18.317868    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:57:18.449664 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:18.449583    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:57:18.543003 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:18.542966    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:57:18.546529 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:18.546504    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-jgkzv"]
Jul 24 16:57:18.549870 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:18.549853    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-jgkzv"
Jul 24 16:57:18.552552 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:18.552534    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-bfwxb\""
Jul 24 16:57:18.552625 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:18.552536    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 24 16:57:18.561211 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:18.561189    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-jgkzv"]
Jul 24 16:57:18.583334 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:18.583305    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-npzx6"]
Jul 24 16:57:18.586454 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:18.586438    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-npzx6"
Jul 24 16:57:18.601994 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:18.601975    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-npzx6"]
Jul 24 16:57:18.637566 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:18.637539    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/497455f7-9ae1-499a-be43-e74bb3cc5fc3-maas-api-tls\") pod \"maas-api-67c5c4478-779b7\" (UID: \"497455f7-9ae1-499a-be43-e74bb3cc5fc3\") " pod="odh-ai-gateway-infra/maas-api-67c5c4478-779b7"
Jul 24 16:57:18.637671 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:18.637601    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2a197f9-b38b-4ac6-98d0-d361f04dfe9b-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-jgkzv\" (UID: \"b2a197f9-b38b-4ac6-98d0-d361f04dfe9b\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-jgkzv"
Jul 24 16:57:18.637671 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:18.637619    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6bcg\" (UniqueName: \"kubernetes.io/projected/b2a197f9-b38b-4ac6-98d0-d361f04dfe9b-kube-api-access-h6bcg\") pod \"payload-pre-processing-687dbb6cf6-jgkzv\" (UID: \"b2a197f9-b38b-4ac6-98d0-d361f04dfe9b\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-jgkzv"
Jul 24 16:57:18.637671 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:18.637642    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/669adb99-c557-4f65-95e0-af083882951f-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-npzx6\" (UID: \"669adb99-c557-4f65-95e0-af083882951f\") " pod="openshift-ingress/payload-processing-5cc6cd444b-npzx6"
Jul 24 16:57:18.637671 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:18.637664    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw975\" (UniqueName: \"kubernetes.io/projected/669adb99-c557-4f65-95e0-af083882951f-kube-api-access-dw975\") pod \"payload-processing-5cc6cd444b-npzx6\" (UID: \"669adb99-c557-4f65-95e0-af083882951f\") " pod="openshift-ingress/payload-processing-5cc6cd444b-npzx6"
Jul 24 16:57:18.639998 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:18.639975    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/497455f7-9ae1-499a-be43-e74bb3cc5fc3-maas-api-tls\") pod \"maas-api-67c5c4478-779b7\" (UID: \"497455f7-9ae1-499a-be43-e74bb3cc5fc3\") " pod="odh-ai-gateway-infra/maas-api-67c5c4478-779b7"
Jul 24 16:57:18.738700 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:18.738616    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2a197f9-b38b-4ac6-98d0-d361f04dfe9b-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-jgkzv\" (UID: \"b2a197f9-b38b-4ac6-98d0-d361f04dfe9b\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-jgkzv"
Jul 24 16:57:18.738700 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:18.738663    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6bcg\" (UniqueName: \"kubernetes.io/projected/b2a197f9-b38b-4ac6-98d0-d361f04dfe9b-kube-api-access-h6bcg\") pod \"payload-pre-processing-687dbb6cf6-jgkzv\" (UID: \"b2a197f9-b38b-4ac6-98d0-d361f04dfe9b\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-jgkzv"
Jul 24 16:57:18.738700 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:18.738699    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/669adb99-c557-4f65-95e0-af083882951f-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-npzx6\" (UID: \"669adb99-c557-4f65-95e0-af083882951f\") " pod="openshift-ingress/payload-processing-5cc6cd444b-npzx6"
Jul 24 16:57:18.739028 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:18.738735    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw975\" (UniqueName: \"kubernetes.io/projected/669adb99-c557-4f65-95e0-af083882951f-kube-api-access-dw975\") pod \"payload-processing-5cc6cd444b-npzx6\" (UID: \"669adb99-c557-4f65-95e0-af083882951f\") " pod="openshift-ingress/payload-processing-5cc6cd444b-npzx6"
Jul 24 16:57:18.739459 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:18.739432    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/669adb99-c557-4f65-95e0-af083882951f-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-npzx6\" (UID: \"669adb99-c557-4f65-95e0-af083882951f\") " pod="openshift-ingress/payload-processing-5cc6cd444b-npzx6"
Jul 24 16:57:18.739543 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:18.739435    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2a197f9-b38b-4ac6-98d0-d361f04dfe9b-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-jgkzv\" (UID: \"b2a197f9-b38b-4ac6-98d0-d361f04dfe9b\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-jgkzv"
Jul 24 16:57:18.756137 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:18.756113    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw975\" (UniqueName: \"kubernetes.io/projected/669adb99-c557-4f65-95e0-af083882951f-kube-api-access-dw975\") pod \"payload-processing-5cc6cd444b-npzx6\" (UID: \"669adb99-c557-4f65-95e0-af083882951f\") " pod="openshift-ingress/payload-processing-5cc6cd444b-npzx6"
Jul 24 16:57:18.756706 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:18.756689    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6bcg\" (UniqueName: \"kubernetes.io/projected/b2a197f9-b38b-4ac6-98d0-d361f04dfe9b-kube-api-access-h6bcg\") pod \"payload-pre-processing-687dbb6cf6-jgkzv\" (UID: \"b2a197f9-b38b-4ac6-98d0-d361f04dfe9b\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-jgkzv"
Jul 24 16:57:18.860850 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:18.860816    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-jgkzv"
Jul 24 16:57:18.886606 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:18.886579    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67c5c4478-779b7"
Jul 24 16:57:18.895385 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:18.895352    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-npzx6"
Jul 24 16:57:19.017885 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:19.017667    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-jgkzv"]
Jul 24 16:57:19.020876 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:57:19.020850    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2a197f9_b38b_4ac6_98d0_d361f04dfe9b.slice/crio-5a27f77c2cf371614b9d356052297212d35e629738c3c29c76c5f4c4d426e2f3 WatchSource:0}: Error finding container 5a27f77c2cf371614b9d356052297212d35e629738c3c29c76c5f4c4d426e2f3: Status 404 returned error can't find the container with id 5a27f77c2cf371614b9d356052297212d35e629738c3c29c76c5f4c4d426e2f3
Jul 24 16:57:19.050146 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:19.050118    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-67c5c4478-779b7"]
Jul 24 16:57:19.053846 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:57:19.053818    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod497455f7_9ae1_499a_be43_e74bb3cc5fc3.slice/crio-10012ba4909bfa9d8666cdf5f317fd08fa5a05cf90f387e778ea188bbc79d405 WatchSource:0}: Error finding container 10012ba4909bfa9d8666cdf5f317fd08fa5a05cf90f387e778ea188bbc79d405: Status 404 returned error can't find the container with id 10012ba4909bfa9d8666cdf5f317fd08fa5a05cf90f387e778ea188bbc79d405
Jul 24 16:57:19.074428 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:19.074402    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-npzx6"]
Jul 24 16:57:19.077600 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:57:19.077575    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod669adb99_c557_4f65_95e0_af083882951f.slice/crio-875f4b167f0514c38fdd2d94f51fb6d439c2e1c8e3024f23453789cb12d5d645 WatchSource:0}: Error finding container 875f4b167f0514c38fdd2d94f51fb6d439c2e1c8e3024f23453789cb12d5d645: Status 404 returned error can't find the container with id 875f4b167f0514c38fdd2d94f51fb6d439c2e1c8e3024f23453789cb12d5d645
Jul 24 16:57:19.374068 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:19.373978    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-jgkzv" event={"ID":"b2a197f9-b38b-4ac6-98d0-d361f04dfe9b","Type":"ContainerStarted","Data":"5a27f77c2cf371614b9d356052297212d35e629738c3c29c76c5f4c4d426e2f3"}
Jul 24 16:57:19.375231 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:19.375208    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-npzx6" event={"ID":"669adb99-c557-4f65-95e0-af083882951f","Type":"ContainerStarted","Data":"875f4b167f0514c38fdd2d94f51fb6d439c2e1c8e3024f23453789cb12d5d645"}
Jul 24 16:57:19.376315 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:19.376295    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67c5c4478-779b7" event={"ID":"497455f7-9ae1-499a-be43-e74bb3cc5fc3","Type":"ContainerStarted","Data":"10012ba4909bfa9d8666cdf5f317fd08fa5a05cf90f387e778ea188bbc79d405"}
Jul 24 16:57:21.395195 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:21.395154    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-npzx6" event={"ID":"669adb99-c557-4f65-95e0-af083882951f","Type":"ContainerStarted","Data":"57781a686dba077f90f3202428789811a04e9c7e0cf6a86d0bfe47bc3cbd3241"}
Jul 24 16:57:21.395652 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:21.395266    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-npzx6"
Jul 24 16:57:21.396858 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:21.396832    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67c5c4478-779b7" event={"ID":"497455f7-9ae1-499a-be43-e74bb3cc5fc3","Type":"ContainerStarted","Data":"77d7dfd69def2aae8aeb79aa1992c7dabbb204825295dbdbf1971d3c0b4da4fb"}
Jul 24 16:57:21.397026 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:21.396897    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-67c5c4478-779b7"
Jul 24 16:57:21.398286 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:21.398263    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-jgkzv" event={"ID":"b2a197f9-b38b-4ac6-98d0-d361f04dfe9b","Type":"ContainerStarted","Data":"b81a6129a318f828d4099cf67d3199b9885aa1c8b4a36c471aee2619216053aa"}
Jul 24 16:57:21.398411 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:21.398391    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-jgkzv"
Jul 24 16:57:21.410722 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:21.410672    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-npzx6" podStartSLOduration=1.2248859460000001 podStartE2EDuration="3.410661001s" totalImagesPullingTime="2.185775055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:57:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:57:21.409566848 +0000 UTC m=+864.051530733" watchObservedRunningTime="2026-07-24 16:57:21.410661001 +0000 UTC m=+864.052624885"
Jul 24 16:57:21.426066 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:21.426022    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-jgkzv" podStartSLOduration=1.194016377 podStartE2EDuration="3.426006079s" totalImagesPullingTime="2.231989702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:57:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:57:21.423802367 +0000 UTC m=+864.065766271" watchObservedRunningTime="2026-07-24 16:57:21.426006079 +0000 UTC m=+864.067969963"
Jul 24 16:57:21.441900 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:21.441851    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-67c5c4478-779b7" podStartSLOduration=2.24242916 podStartE2EDuration="4.441835283s" totalImagesPullingTime="2.199406123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:57:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:57:21.440004689 +0000 UTC m=+864.081968595" watchObservedRunningTime="2026-07-24 16:57:21.441835283 +0000 UTC m=+864.083799167"
Jul 24 16:57:27.409158 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:27.409132    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-67c5c4478-779b7"
Jul 24 16:57:28.364177 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:28.364139    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:57:28.464681 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:28.464647    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:57:32.405189 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:32.405140    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-jgkzv" podUID="b2a197f9-b38b-4ac6-98d0-d361f04dfe9b" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.37:9004: connect: connection refused"
Jul 24 16:57:32.405620 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:32.405514    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-npzx6"
Jul 24 16:57:41.431200 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:41.431164    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:57:42.404127 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:42.404092    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-jgkzv"
Jul 24 16:57:57.994348 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:57.994320    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ddklx_fb5a9838-1e05-4f86-b46c-760bd7aaa038/console-operator/1.log"
Jul 24 16:57:57.994803 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:57:57.994601    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ddklx_fb5a9838-1e05-4f86-b46c-760bd7aaa038/console-operator/1.log"
Jul 24 16:58:07.288714 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:07.288677    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-6lzp5"]
Jul 24 16:58:07.292340 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:07.292316    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6lzp5"
Jul 24 16:58:07.294738 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:07.294714    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-s45jh\""
Jul 24 16:58:07.294871 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:07.294719    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 24 16:58:07.294871 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:07.294718    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 24 16:58:07.294871 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:07.294787    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 24 16:58:07.302127 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:07.302091    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-6lzp5"]
Jul 24 16:58:07.316860 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:07.316831    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:58:07.355082 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:07.355045    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2db5q\" (UniqueName: \"kubernetes.io/projected/bf8bdd3a-7faa-4a02-934a-b25c232ba6fe-kube-api-access-2db5q\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6lzp5\" (UID: \"bf8bdd3a-7faa-4a02-934a-b25c232ba6fe\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6lzp5"
Jul 24 16:58:07.355357 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:07.355216    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf8bdd3a-7faa-4a02-934a-b25c232ba6fe-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6lzp5\" (UID: \"bf8bdd3a-7faa-4a02-934a-b25c232ba6fe\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6lzp5"
Jul 24 16:58:07.355357 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:07.355261    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf8bdd3a-7faa-4a02-934a-b25c232ba6fe-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6lzp5\" (UID: \"bf8bdd3a-7faa-4a02-934a-b25c232ba6fe\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6lzp5"
Jul 24 16:58:07.355357 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:07.355306    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf8bdd3a-7faa-4a02-934a-b25c232ba6fe-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6lzp5\" (UID: \"bf8bdd3a-7faa-4a02-934a-b25c232ba6fe\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6lzp5"
Jul 24 16:58:07.355357 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:07.355338    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf8bdd3a-7faa-4a02-934a-b25c232ba6fe-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6lzp5\" (UID: \"bf8bdd3a-7faa-4a02-934a-b25c232ba6fe\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6lzp5"
Jul 24 16:58:07.355615 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:07.355376    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf8bdd3a-7faa-4a02-934a-b25c232ba6fe-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6lzp5\" (UID: \"bf8bdd3a-7faa-4a02-934a-b25c232ba6fe\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6lzp5"
Jul 24 16:58:07.455811 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:07.455780    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf8bdd3a-7faa-4a02-934a-b25c232ba6fe-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6lzp5\" (UID: \"bf8bdd3a-7faa-4a02-934a-b25c232ba6fe\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6lzp5"
Jul 24 16:58:07.455811 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:07.455814    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf8bdd3a-7faa-4a02-934a-b25c232ba6fe-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6lzp5\" (UID: \"bf8bdd3a-7faa-4a02-934a-b25c232ba6fe\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6lzp5"
Jul 24 16:58:07.456043 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:07.455849    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf8bdd3a-7faa-4a02-934a-b25c232ba6fe-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6lzp5\" (UID: \"bf8bdd3a-7faa-4a02-934a-b25c232ba6fe\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6lzp5"
Jul 24 16:58:07.456043 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:07.455879    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2db5q\" (UniqueName: \"kubernetes.io/projected/bf8bdd3a-7faa-4a02-934a-b25c232ba6fe-kube-api-access-2db5q\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6lzp5\" (UID: \"bf8bdd3a-7faa-4a02-934a-b25c232ba6fe\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6lzp5"
Jul 24 16:58:07.456043 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:07.455986    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf8bdd3a-7faa-4a02-934a-b25c232ba6fe-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6lzp5\" (UID: \"bf8bdd3a-7faa-4a02-934a-b25c232ba6fe\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6lzp5"
Jul 24 16:58:07.456043 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:07.456009    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf8bdd3a-7faa-4a02-934a-b25c232ba6fe-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6lzp5\" (UID: \"bf8bdd3a-7faa-4a02-934a-b25c232ba6fe\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6lzp5"
Jul 24 16:58:07.456307 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:07.456281    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf8bdd3a-7faa-4a02-934a-b25c232ba6fe-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6lzp5\" (UID: \"bf8bdd3a-7faa-4a02-934a-b25c232ba6fe\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6lzp5"
Jul 24 16:58:07.456400 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:07.456319    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf8bdd3a-7faa-4a02-934a-b25c232ba6fe-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6lzp5\" (UID: \"bf8bdd3a-7faa-4a02-934a-b25c232ba6fe\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6lzp5"
Jul 24 16:58:07.456400 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:07.456383    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf8bdd3a-7faa-4a02-934a-b25c232ba6fe-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6lzp5\" (UID: \"bf8bdd3a-7faa-4a02-934a-b25c232ba6fe\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6lzp5"
Jul 24 16:58:07.458367 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:07.458339    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf8bdd3a-7faa-4a02-934a-b25c232ba6fe-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6lzp5\" (UID: \"bf8bdd3a-7faa-4a02-934a-b25c232ba6fe\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6lzp5"
Jul 24 16:58:07.458707 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:07.458685    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf8bdd3a-7faa-4a02-934a-b25c232ba6fe-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6lzp5\" (UID: \"bf8bdd3a-7faa-4a02-934a-b25c232ba6fe\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6lzp5"
Jul 24 16:58:07.466352 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:07.466317    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2db5q\" (UniqueName: \"kubernetes.io/projected/bf8bdd3a-7faa-4a02-934a-b25c232ba6fe-kube-api-access-2db5q\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6lzp5\" (UID: \"bf8bdd3a-7faa-4a02-934a-b25c232ba6fe\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6lzp5"
Jul 24 16:58:07.603136 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:07.603047    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6lzp5"
Jul 24 16:58:07.742122 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:07.742092    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-6lzp5"]
Jul 24 16:58:07.745545 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:58:07.745517    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf8bdd3a_7faa_4a02_934a_b25c232ba6fe.slice/crio-48123ebb3cb4dcca2914de78f537929277f801c114ed9ff5de2a704686211126 WatchSource:0}: Error finding container 48123ebb3cb4dcca2914de78f537929277f801c114ed9ff5de2a704686211126: Status 404 returned error can't find the container with id 48123ebb3cb4dcca2914de78f537929277f801c114ed9ff5de2a704686211126
Jul 24 16:58:08.632711 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:08.632520    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6lzp5" event={"ID":"bf8bdd3a-7faa-4a02-934a-b25c232ba6fe","Type":"ContainerStarted","Data":"48123ebb3cb4dcca2914de78f537929277f801c114ed9ff5de2a704686211126"}
Jul 24 16:58:09.639066 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:09.639029    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6lzp5" event={"ID":"bf8bdd3a-7faa-4a02-934a-b25c232ba6fe","Type":"ContainerStarted","Data":"1a93ea9663cf9994439dfbc7cfeaf330734a42e07130c13740b1e32014903aa2"}
Jul 24 16:58:09.656909 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:09.656853    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6lzp5" podStartSLOduration=1.704045536 podStartE2EDuration="2.656835369s" totalImagesPullingTime="952.789833ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:58:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:58:09.65490113 +0000 UTC m=+912.296865040" watchObservedRunningTime="2026-07-24 16:58:09.656835369 +0000 UTC m=+912.298799304"
Jul 24 16:58:13.685189 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:13.685150    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-m2x4g"]
Jul 24 16:58:13.688858 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:13.688836    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m2x4g"
Jul 24 16:58:13.691212 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:13.691191    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 24 16:58:13.698335 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:13.698312    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-m2x4g"]
Jul 24 16:58:13.815499 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:13.815457    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1633363-139f-4c13-a590-7fb2d8de28e5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m2x4g\" (UID: \"d1633363-139f-4c13-a590-7fb2d8de28e5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m2x4g"
Jul 24 16:58:13.815499 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:13.815502    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zt67\" (UniqueName: \"kubernetes.io/projected/d1633363-139f-4c13-a590-7fb2d8de28e5-kube-api-access-4zt67\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m2x4g\" (UID: \"d1633363-139f-4c13-a590-7fb2d8de28e5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m2x4g"
Jul 24 16:58:13.815733 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:13.815601    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1633363-139f-4c13-a590-7fb2d8de28e5-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m2x4g\" (UID: \"d1633363-139f-4c13-a590-7fb2d8de28e5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m2x4g"
Jul 24 16:58:13.815733 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:13.815648    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1633363-139f-4c13-a590-7fb2d8de28e5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m2x4g\" (UID: \"d1633363-139f-4c13-a590-7fb2d8de28e5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m2x4g"
Jul 24 16:58:13.815733 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:13.815671    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1633363-139f-4c13-a590-7fb2d8de28e5-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m2x4g\" (UID: \"d1633363-139f-4c13-a590-7fb2d8de28e5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m2x4g"
Jul 24 16:58:13.815733 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:13.815687    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1633363-139f-4c13-a590-7fb2d8de28e5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m2x4g\" (UID: \"d1633363-139f-4c13-a590-7fb2d8de28e5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m2x4g"
Jul 24 16:58:13.890200 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:13.890166    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:58:13.917059 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:13.917023    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1633363-139f-4c13-a590-7fb2d8de28e5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m2x4g\" (UID: \"d1633363-139f-4c13-a590-7fb2d8de28e5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m2x4g"
Jul 24 16:58:13.917059 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:13.917059    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zt67\" (UniqueName: \"kubernetes.io/projected/d1633363-139f-4c13-a590-7fb2d8de28e5-kube-api-access-4zt67\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m2x4g\" (UID: \"d1633363-139f-4c13-a590-7fb2d8de28e5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m2x4g"
Jul 24 16:58:13.917311 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:13.917092    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1633363-139f-4c13-a590-7fb2d8de28e5-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m2x4g\" (UID: \"d1633363-139f-4c13-a590-7fb2d8de28e5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m2x4g"
Jul 24 16:58:13.917311 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:13.917120    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1633363-139f-4c13-a590-7fb2d8de28e5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m2x4g\" (UID: \"d1633363-139f-4c13-a590-7fb2d8de28e5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m2x4g"
Jul 24 16:58:13.917311 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:13.917146    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1633363-139f-4c13-a590-7fb2d8de28e5-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m2x4g\" (UID: \"d1633363-139f-4c13-a590-7fb2d8de28e5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m2x4g"
Jul 24 16:58:13.917311 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:13.917161    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1633363-139f-4c13-a590-7fb2d8de28e5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m2x4g\" (UID: \"d1633363-139f-4c13-a590-7fb2d8de28e5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m2x4g"
Jul 24 16:58:13.917562 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:13.917528    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1633363-139f-4c13-a590-7fb2d8de28e5-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m2x4g\" (UID: \"d1633363-139f-4c13-a590-7fb2d8de28e5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m2x4g"
Jul 24 16:58:13.917682 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:13.917538    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1633363-139f-4c13-a590-7fb2d8de28e5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m2x4g\" (UID: \"d1633363-139f-4c13-a590-7fb2d8de28e5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m2x4g"
Jul 24 16:58:13.917682 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:13.917584    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1633363-139f-4c13-a590-7fb2d8de28e5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m2x4g\" (UID: \"d1633363-139f-4c13-a590-7fb2d8de28e5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m2x4g"
Jul 24 16:58:13.919454 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:13.919428    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1633363-139f-4c13-a590-7fb2d8de28e5-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m2x4g\" (UID: \"d1633363-139f-4c13-a590-7fb2d8de28e5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m2x4g"
Jul 24 16:58:13.919720 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:13.919700    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1633363-139f-4c13-a590-7fb2d8de28e5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m2x4g\" (UID: \"d1633363-139f-4c13-a590-7fb2d8de28e5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m2x4g"
Jul 24 16:58:13.924756 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:13.924732    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zt67\" (UniqueName: \"kubernetes.io/projected/d1633363-139f-4c13-a590-7fb2d8de28e5-kube-api-access-4zt67\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m2x4g\" (UID: \"d1633363-139f-4c13-a590-7fb2d8de28e5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m2x4g"
Jul 24 16:58:13.999799 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:13.999709    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m2x4g"
Jul 24 16:58:14.133406 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:14.133376    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-m2x4g"]
Jul 24 16:58:14.135477 ip-10-0-133-234 kubenswrapper[2563]: W0724 16:58:14.135452    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1633363_139f_4c13_a590_7fb2d8de28e5.slice/crio-821d4a7fef7097fdd71f8e9039b1158ef2629304b14dacc263ea01779d930d4b WatchSource:0}: Error finding container 821d4a7fef7097fdd71f8e9039b1158ef2629304b14dacc263ea01779d930d4b: Status 404 returned error can't find the container with id 821d4a7fef7097fdd71f8e9039b1158ef2629304b14dacc263ea01779d930d4b
Jul 24 16:58:14.664308 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:14.664274    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m2x4g" event={"ID":"d1633363-139f-4c13-a590-7fb2d8de28e5","Type":"ContainerStarted","Data":"56dc5fb51a167e4191943667463ab38e5e2a2883aee694041d5eed62cfd36ff0"}
Jul 24 16:58:14.664512 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:14.664324    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m2x4g" event={"ID":"d1633363-139f-4c13-a590-7fb2d8de28e5","Type":"ContainerStarted","Data":"821d4a7fef7097fdd71f8e9039b1158ef2629304b14dacc263ea01779d930d4b"}
Jul 24 16:58:14.682205 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:14.682152    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m2x4g" podStartSLOduration=1.5120126489999999 podStartE2EDuration="1.682134869s" totalImagesPullingTime="170.12222ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:58:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:58:14.6802363 +0000 UTC m=+917.322200203" watchObservedRunningTime="2026-07-24 16:58:14.682134869 +0000 UTC m=+917.324098753"
Jul 24 16:58:17.603204 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:17.603164    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6lzp5"
Jul 24 16:58:17.603642 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:17.603250    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6lzp5"
Jul 24 16:58:17.615875 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:17.615851    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6lzp5"
Jul 24 16:58:17.692414 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:17.692388    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6lzp5"
Jul 24 16:58:18.714926 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:18.714891    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:58:24.002627 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:24.002590    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m2x4g"
Jul 24 16:58:24.003036 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:24.002650    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m2x4g"
Jul 24 16:58:24.012686 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:24.012660    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m2x4g"
Jul 24 16:58:24.724679 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:24.724643    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m2x4g"
Jul 24 16:58:25.746057 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:25.746020    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:58:33.907544 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:58:33.907511    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:59:21.913587 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:59:21.913507    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:59:22.016428 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:59:22.016389    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:59:23.009191 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:59:23.009162    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 16:59:23.109566 ip-10-0-133-234 kubenswrapper[2563]: I0724 16:59:23.109532    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:00:00.141455 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:00:00.141421    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748540-7s96c"]
Jul 24 17:00:00.145391 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:00:00.145366    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748540-7s96c"
Jul 24 17:00:00.149692 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:00:00.149012    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jdhj4\""
Jul 24 17:00:00.150770 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:00:00.150664    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748540-7s96c"]
Jul 24 17:00:00.161214 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:00:00.161192    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748540-7s96c"
Jul 24 17:00:00.505436 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:00:00.505409    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748540-7s96c"]
Jul 24 17:00:00.507850 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:00:00.507822    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44441fd3_52da_4fe3_975c_60dd4af20404.slice/crio-e165460dc408ca44c1bb96258a934ac894227ba86d1a1ca4400e3a1c718d032c WatchSource:0}: Error finding container e165460dc408ca44c1bb96258a934ac894227ba86d1a1ca4400e3a1c718d032c: Status 404 returned error can't find the container with id e165460dc408ca44c1bb96258a934ac894227ba86d1a1ca4400e3a1c718d032c
Jul 24 17:00:00.509809 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:00:00.509794    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 17:00:01.159140 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:00:01.159106    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748540-7s96c" event={"ID":"44441fd3-52da-4fe3-975c-60dd4af20404","Type":"ContainerStarted","Data":"e165460dc408ca44c1bb96258a934ac894227ba86d1a1ca4400e3a1c718d032c"}
Jul 24 17:00:08.196003 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:00:08.195965    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748540-7s96c" event={"ID":"44441fd3-52da-4fe3-975c-60dd4af20404","Type":"ContainerStarted","Data":"fa40f096fa9d4ddaec52822972433ecc1a08e7def6c384443c0964507d496892"}
Jul 24 17:00:08.213083 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:00:08.213000    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748540-7s96c" podStartSLOduration=1.087268823 podStartE2EDuration="8.212986424s" totalImagesPullingTime="7.125717601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:00:08.210866275 +0000 UTC m=+1030.852830186" watchObservedRunningTime="2026-07-24 17:00:08.212986424 +0000 UTC m=+1030.854950307"
Jul 24 17:00:09.202417 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:00:09.202380    2563 generic.go:356] "Generic (PLEG): container finished" podID="44441fd3-52da-4fe3-975c-60dd4af20404" containerID="fa40f096fa9d4ddaec52822972433ecc1a08e7def6c384443c0964507d496892" exitCode=0
Jul 24 17:00:09.202417 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:00:09.202420    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748540-7s96c" event={"ID":"44441fd3-52da-4fe3-975c-60dd4af20404","Type":"ContainerDied","Data":"fa40f096fa9d4ddaec52822972433ecc1a08e7def6c384443c0964507d496892"}
Jul 24 17:00:09.243794 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:00:09.243765    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7d8c754f6b-npwxj"]
Jul 24 17:00:09.243976 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:00:09.243951    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7d8c754f6b-npwxj" podUID="b56649dc-ecfc-4258-885b-484147067106" containerName="manager" containerID="cri-o://875c9c33551da006a3bab55aaa1d34fe15001143c85125530d898c0ce357af19" gracePeriod=30
Jul 24 17:00:09.492028 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:00:09.492001    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d8c754f6b-npwxj"
Jul 24 17:00:09.626315 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:00:09.626283    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkfng\" (UniqueName: \"kubernetes.io/projected/b56649dc-ecfc-4258-885b-484147067106-kube-api-access-zkfng\") pod \"b56649dc-ecfc-4258-885b-484147067106\" (UID: \"b56649dc-ecfc-4258-885b-484147067106\") "
Jul 24 17:00:09.626485 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:00:09.626345    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b56649dc-ecfc-4258-885b-484147067106-webhook-cert\") pod \"b56649dc-ecfc-4258-885b-484147067106\" (UID: \"b56649dc-ecfc-4258-885b-484147067106\") "
Jul 24 17:00:09.628380 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:00:09.628355    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b56649dc-ecfc-4258-885b-484147067106-kube-api-access-zkfng" (OuterVolumeSpecName: "kube-api-access-zkfng") pod "b56649dc-ecfc-4258-885b-484147067106" (UID: "b56649dc-ecfc-4258-885b-484147067106"). InnerVolumeSpecName "kube-api-access-zkfng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:00:09.628445 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:00:09.628359    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b56649dc-ecfc-4258-885b-484147067106-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b56649dc-ecfc-4258-885b-484147067106" (UID: "b56649dc-ecfc-4258-885b-484147067106"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:00:09.727175 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:00:09.727092    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkfng\" (UniqueName: \"kubernetes.io/projected/b56649dc-ecfc-4258-885b-484147067106-kube-api-access-zkfng\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:00:09.727175 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:00:09.727120    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b56649dc-ecfc-4258-885b-484147067106-webhook-cert\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:00:10.208036 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:00:10.207999    2563 generic.go:356] "Generic (PLEG): container finished" podID="b56649dc-ecfc-4258-885b-484147067106" containerID="875c9c33551da006a3bab55aaa1d34fe15001143c85125530d898c0ce357af19" exitCode=0
Jul 24 17:00:10.208036 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:00:10.208040    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d8c754f6b-npwxj"
Jul 24 17:00:10.208560 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:00:10.208086    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d8c754f6b-npwxj" event={"ID":"b56649dc-ecfc-4258-885b-484147067106","Type":"ContainerDied","Data":"875c9c33551da006a3bab55aaa1d34fe15001143c85125530d898c0ce357af19"}
Jul 24 17:00:10.208560 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:00:10.208145    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d8c754f6b-npwxj" event={"ID":"b56649dc-ecfc-4258-885b-484147067106","Type":"ContainerDied","Data":"3b6ee874f0fb829a94a428c90dd167f88bed0372a10aa6fe7d2c323b85e50d8a"}
Jul 24 17:00:10.208560 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:00:10.208179    2563 scope.go:117] "RemoveContainer" containerID="875c9c33551da006a3bab55aaa1d34fe15001143c85125530d898c0ce357af19"
Jul 24 17:00:10.219248 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:00:10.219231    2563 scope.go:117] "RemoveContainer" containerID="875c9c33551da006a3bab55aaa1d34fe15001143c85125530d898c0ce357af19"
Jul 24 17:00:10.219529 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:00:10.219503    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"875c9c33551da006a3bab55aaa1d34fe15001143c85125530d898c0ce357af19\": container with ID starting with 875c9c33551da006a3bab55aaa1d34fe15001143c85125530d898c0ce357af19 not found: ID does not exist" containerID="875c9c33551da006a3bab55aaa1d34fe15001143c85125530d898c0ce357af19"
Jul 24 17:00:10.219587 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:00:10.219539    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"875c9c33551da006a3bab55aaa1d34fe15001143c85125530d898c0ce357af19"} err="failed to get container status \"875c9c33551da006a3bab55aaa1d34fe15001143c85125530d898c0ce357af19\": rpc error: code = NotFound desc = could not find container \"875c9c33551da006a3bab55aaa1d34fe15001143c85125530d898c0ce357af19\": container with ID starting with 875c9c33551da006a3bab55aaa1d34fe15001143c85125530d898c0ce357af19 not found: ID does not exist"
Jul 24 17:00:10.225416 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:00:10.225394    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7d8c754f6b-npwxj"]
Jul 24 17:00:10.232373 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:00:10.232349    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7d8c754f6b-npwxj"]
Jul 24 17:00:10.347268 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:00:10.347245    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748540-7s96c"
Jul 24 17:00:11.214842 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:00:11.214806    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748540-7s96c" event={"ID":"44441fd3-52da-4fe3-975c-60dd4af20404","Type":"ContainerDied","Data":"e165460dc408ca44c1bb96258a934ac894227ba86d1a1ca4400e3a1c718d032c"}
Jul 24 17:00:11.215346 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:00:11.214861    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e165460dc408ca44c1bb96258a934ac894227ba86d1a1ca4400e3a1c718d032c"
Jul 24 17:00:11.215346 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:00:11.214814    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748540-7s96c"
Jul 24 17:00:12.002642 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:00:12.002609    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b56649dc-ecfc-4258-885b-484147067106" path="/var/lib/kubelet/pods/b56649dc-ecfc-4258-885b-484147067106/volumes"
Jul 24 17:01:08.824390 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:08.824360    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:01:11.033638 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:11.033602    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7d8c754f6b-n2rjm"]
Jul 24 17:01:11.034034 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:11.033958    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44441fd3-52da-4fe3-975c-60dd4af20404" containerName="cleanup"
Jul 24 17:01:11.034034 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:11.033969    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="44441fd3-52da-4fe3-975c-60dd4af20404" containerName="cleanup"
Jul 24 17:01:11.034034 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:11.033982    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b56649dc-ecfc-4258-885b-484147067106" containerName="manager"
Jul 24 17:01:11.034034 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:11.033988    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b56649dc-ecfc-4258-885b-484147067106" containerName="manager"
Jul 24 17:01:11.034165 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:11.034048    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b56649dc-ecfc-4258-885b-484147067106" containerName="manager"
Jul 24 17:01:11.034165 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:11.034056    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="44441fd3-52da-4fe3-975c-60dd4af20404" containerName="cleanup"
Jul 24 17:01:11.038393 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:11.038372    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d8c754f6b-n2rjm"
Jul 24 17:01:11.041818 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:11.041799    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 24 17:01:11.041918 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:11.041803    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-6x8jt\""
Jul 24 17:01:11.041918 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:11.041807    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 24 17:01:11.046444 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:11.046424    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d8c754f6b-n2rjm"]
Jul 24 17:01:11.049539 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:11.049520    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6df3bebd-0a1b-497b-bfb2-5bb85067365f-webhook-cert\") pod \"maas-controller-7d8c754f6b-n2rjm\" (UID: \"6df3bebd-0a1b-497b-bfb2-5bb85067365f\") " pod="opendatahub/maas-controller-7d8c754f6b-n2rjm"
Jul 24 17:01:11.049625 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:11.049596    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lgx4\" (UniqueName: \"kubernetes.io/projected/6df3bebd-0a1b-497b-bfb2-5bb85067365f-kube-api-access-8lgx4\") pod \"maas-controller-7d8c754f6b-n2rjm\" (UID: \"6df3bebd-0a1b-497b-bfb2-5bb85067365f\") " pod="opendatahub/maas-controller-7d8c754f6b-n2rjm"
Jul 24 17:01:11.150322 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:11.150287    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lgx4\" (UniqueName: \"kubernetes.io/projected/6df3bebd-0a1b-497b-bfb2-5bb85067365f-kube-api-access-8lgx4\") pod \"maas-controller-7d8c754f6b-n2rjm\" (UID: \"6df3bebd-0a1b-497b-bfb2-5bb85067365f\") " pod="opendatahub/maas-controller-7d8c754f6b-n2rjm"
Jul 24 17:01:11.150491 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:11.150337    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6df3bebd-0a1b-497b-bfb2-5bb85067365f-webhook-cert\") pod \"maas-controller-7d8c754f6b-n2rjm\" (UID: \"6df3bebd-0a1b-497b-bfb2-5bb85067365f\") " pod="opendatahub/maas-controller-7d8c754f6b-n2rjm"
Jul 24 17:01:11.152675 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:11.152657    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6df3bebd-0a1b-497b-bfb2-5bb85067365f-webhook-cert\") pod \"maas-controller-7d8c754f6b-n2rjm\" (UID: \"6df3bebd-0a1b-497b-bfb2-5bb85067365f\") " pod="opendatahub/maas-controller-7d8c754f6b-n2rjm"
Jul 24 17:01:11.158730 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:11.158705    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lgx4\" (UniqueName: \"kubernetes.io/projected/6df3bebd-0a1b-497b-bfb2-5bb85067365f-kube-api-access-8lgx4\") pod \"maas-controller-7d8c754f6b-n2rjm\" (UID: \"6df3bebd-0a1b-497b-bfb2-5bb85067365f\") " pod="opendatahub/maas-controller-7d8c754f6b-n2rjm"
Jul 24 17:01:11.349848 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:11.349762    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d8c754f6b-n2rjm"
Jul 24 17:01:11.479788 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:11.479766    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d8c754f6b-n2rjm"]
Jul 24 17:01:11.481653 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:01:11.481623    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df3bebd_0a1b_497b_bfb2_5bb85067365f.slice/crio-b52e0be7f8d70c5bb26cfb51017c20cb79b8de9d92395dd913dfcb8c93ce1e29 WatchSource:0}: Error finding container b52e0be7f8d70c5bb26cfb51017c20cb79b8de9d92395dd913dfcb8c93ce1e29: Status 404 returned error can't find the container with id b52e0be7f8d70c5bb26cfb51017c20cb79b8de9d92395dd913dfcb8c93ce1e29
Jul 24 17:01:11.505245 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:11.505216    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d8c754f6b-n2rjm" event={"ID":"6df3bebd-0a1b-497b-bfb2-5bb85067365f","Type":"ContainerStarted","Data":"b52e0be7f8d70c5bb26cfb51017c20cb79b8de9d92395dd913dfcb8c93ce1e29"}
Jul 24 17:01:12.511460 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:12.511424    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d8c754f6b-n2rjm" event={"ID":"6df3bebd-0a1b-497b-bfb2-5bb85067365f","Type":"ContainerStarted","Data":"3bb58a8ed18aef330f7034805c3f5c63b2060641e0aa5d1cfbe2dc3cd4c652e4"}
Jul 24 17:01:12.511831 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:12.511543    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7d8c754f6b-n2rjm"
Jul 24 17:01:12.528412 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:12.528366    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7d8c754f6b-n2rjm" podStartSLOduration=1.115409516 podStartE2EDuration="1.528352089s" totalImagesPullingTime="412.942573ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:01:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:01:12.526556842 +0000 UTC m=+1095.168520749" watchObservedRunningTime="2026-07-24 17:01:12.528352089 +0000 UTC m=+1095.170315973"
Jul 24 17:01:23.519069 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:23.519035    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7d8c754f6b-n2rjm"
Jul 24 17:01:28.061470 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:28.061431    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:01:28.200784 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:28.200750    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7d8c754f6b-n2rjm"]
Jul 24 17:01:28.200984 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:28.200947    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7d8c754f6b-n2rjm" podUID="6df3bebd-0a1b-497b-bfb2-5bb85067365f" containerName="manager" containerID="cri-o://3bb58a8ed18aef330f7034805c3f5c63b2060641e0aa5d1cfbe2dc3cd4c652e4" gracePeriod=30
Jul 24 17:01:28.254425 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:28.254389    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:01:28.446382 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:28.446353    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d8c754f6b-n2rjm"
Jul 24 17:01:28.492325 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:28.492297    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6df3bebd-0a1b-497b-bfb2-5bb85067365f-webhook-cert\") pod \"6df3bebd-0a1b-497b-bfb2-5bb85067365f\" (UID: \"6df3bebd-0a1b-497b-bfb2-5bb85067365f\") "
Jul 24 17:01:28.492478 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:28.492358    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lgx4\" (UniqueName: \"kubernetes.io/projected/6df3bebd-0a1b-497b-bfb2-5bb85067365f-kube-api-access-8lgx4\") pod \"6df3bebd-0a1b-497b-bfb2-5bb85067365f\" (UID: \"6df3bebd-0a1b-497b-bfb2-5bb85067365f\") "
Jul 24 17:01:28.494384 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:28.494353    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df3bebd-0a1b-497b-bfb2-5bb85067365f-kube-api-access-8lgx4" (OuterVolumeSpecName: "kube-api-access-8lgx4") pod "6df3bebd-0a1b-497b-bfb2-5bb85067365f" (UID: "6df3bebd-0a1b-497b-bfb2-5bb85067365f"). InnerVolumeSpecName "kube-api-access-8lgx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:01:28.494691 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:28.494669    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6df3bebd-0a1b-497b-bfb2-5bb85067365f-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "6df3bebd-0a1b-497b-bfb2-5bb85067365f" (UID: "6df3bebd-0a1b-497b-bfb2-5bb85067365f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:01:28.590187 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:28.590100    2563 generic.go:356] "Generic (PLEG): container finished" podID="6df3bebd-0a1b-497b-bfb2-5bb85067365f" containerID="3bb58a8ed18aef330f7034805c3f5c63b2060641e0aa5d1cfbe2dc3cd4c652e4" exitCode=0
Jul 24 17:01:28.590187 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:28.590142    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d8c754f6b-n2rjm"
Jul 24 17:01:28.590375 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:28.590200    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d8c754f6b-n2rjm" event={"ID":"6df3bebd-0a1b-497b-bfb2-5bb85067365f","Type":"ContainerDied","Data":"3bb58a8ed18aef330f7034805c3f5c63b2060641e0aa5d1cfbe2dc3cd4c652e4"}
Jul 24 17:01:28.590375 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:28.590245    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d8c754f6b-n2rjm" event={"ID":"6df3bebd-0a1b-497b-bfb2-5bb85067365f","Type":"ContainerDied","Data":"b52e0be7f8d70c5bb26cfb51017c20cb79b8de9d92395dd913dfcb8c93ce1e29"}
Jul 24 17:01:28.590375 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:28.590267    2563 scope.go:117] "RemoveContainer" containerID="3bb58a8ed18aef330f7034805c3f5c63b2060641e0aa5d1cfbe2dc3cd4c652e4"
Jul 24 17:01:28.593252 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:28.593236    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6df3bebd-0a1b-497b-bfb2-5bb85067365f-webhook-cert\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:01:28.593332 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:28.593255    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lgx4\" (UniqueName: \"kubernetes.io/projected/6df3bebd-0a1b-497b-bfb2-5bb85067365f-kube-api-access-8lgx4\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:01:28.599362 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:28.599345    2563 scope.go:117] "RemoveContainer" containerID="3bb58a8ed18aef330f7034805c3f5c63b2060641e0aa5d1cfbe2dc3cd4c652e4"
Jul 24 17:01:28.599592 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:01:28.599575    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bb58a8ed18aef330f7034805c3f5c63b2060641e0aa5d1cfbe2dc3cd4c652e4\": container with ID starting with 3bb58a8ed18aef330f7034805c3f5c63b2060641e0aa5d1cfbe2dc3cd4c652e4 not found: ID does not exist" containerID="3bb58a8ed18aef330f7034805c3f5c63b2060641e0aa5d1cfbe2dc3cd4c652e4"
Jul 24 17:01:28.599640 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:28.599600    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bb58a8ed18aef330f7034805c3f5c63b2060641e0aa5d1cfbe2dc3cd4c652e4"} err="failed to get container status \"3bb58a8ed18aef330f7034805c3f5c63b2060641e0aa5d1cfbe2dc3cd4c652e4\": rpc error: code = NotFound desc = could not find container \"3bb58a8ed18aef330f7034805c3f5c63b2060641e0aa5d1cfbe2dc3cd4c652e4\": container with ID starting with 3bb58a8ed18aef330f7034805c3f5c63b2060641e0aa5d1cfbe2dc3cd4c652e4 not found: ID does not exist"
Jul 24 17:01:28.612081 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:28.612052    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7d8c754f6b-n2rjm"]
Jul 24 17:01:28.615736 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:28.615715    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7d8c754f6b-n2rjm"]
Jul 24 17:01:29.464105 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:29.464071    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:01:29.602142 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:29.602108    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7d8c754f6b-qcxmb"]
Jul 24 17:01:29.602634 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:29.602615    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6df3bebd-0a1b-497b-bfb2-5bb85067365f" containerName="manager"
Jul 24 17:01:29.602634 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:29.602634    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6df3bebd-0a1b-497b-bfb2-5bb85067365f" containerName="manager"
Jul 24 17:01:29.602769 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:29.602691    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6df3bebd-0a1b-497b-bfb2-5bb85067365f" containerName="manager"
Jul 24 17:01:29.607449 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:29.607431    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d8c754f6b-qcxmb"
Jul 24 17:01:29.611050 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:29.611030    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 24 17:01:29.611149 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:29.611028    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 24 17:01:29.611149 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:29.611094    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-6x8jt\""
Jul 24 17:01:29.615066 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:29.615046    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d8c754f6b-qcxmb"]
Jul 24 17:01:29.702666 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:29.702627    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6lm2\" (UniqueName: \"kubernetes.io/projected/ce02e79c-6549-4c69-b469-bba26b0567cd-kube-api-access-n6lm2\") pod \"maas-controller-7d8c754f6b-qcxmb\" (UID: \"ce02e79c-6549-4c69-b469-bba26b0567cd\") " pod="opendatahub/maas-controller-7d8c754f6b-qcxmb"
Jul 24 17:01:29.702855 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:29.702746    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce02e79c-6549-4c69-b469-bba26b0567cd-webhook-cert\") pod \"maas-controller-7d8c754f6b-qcxmb\" (UID: \"ce02e79c-6549-4c69-b469-bba26b0567cd\") " pod="opendatahub/maas-controller-7d8c754f6b-qcxmb"
Jul 24 17:01:29.803835 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:29.803751    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce02e79c-6549-4c69-b469-bba26b0567cd-webhook-cert\") pod \"maas-controller-7d8c754f6b-qcxmb\" (UID: \"ce02e79c-6549-4c69-b469-bba26b0567cd\") " pod="opendatahub/maas-controller-7d8c754f6b-qcxmb"
Jul 24 17:01:29.803835 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:29.803800    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6lm2\" (UniqueName: \"kubernetes.io/projected/ce02e79c-6549-4c69-b469-bba26b0567cd-kube-api-access-n6lm2\") pod \"maas-controller-7d8c754f6b-qcxmb\" (UID: \"ce02e79c-6549-4c69-b469-bba26b0567cd\") " pod="opendatahub/maas-controller-7d8c754f6b-qcxmb"
Jul 24 17:01:29.806253 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:29.806233    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce02e79c-6549-4c69-b469-bba26b0567cd-webhook-cert\") pod \"maas-controller-7d8c754f6b-qcxmb\" (UID: \"ce02e79c-6549-4c69-b469-bba26b0567cd\") " pod="opendatahub/maas-controller-7d8c754f6b-qcxmb"
Jul 24 17:01:29.812371 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:29.812346    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6lm2\" (UniqueName: \"kubernetes.io/projected/ce02e79c-6549-4c69-b469-bba26b0567cd-kube-api-access-n6lm2\") pod \"maas-controller-7d8c754f6b-qcxmb\" (UID: \"ce02e79c-6549-4c69-b469-bba26b0567cd\") " pod="opendatahub/maas-controller-7d8c754f6b-qcxmb"
Jul 24 17:01:29.919189 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:29.919153    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d8c754f6b-qcxmb"
Jul 24 17:01:30.004026 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:30.003997    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6df3bebd-0a1b-497b-bfb2-5bb85067365f" path="/var/lib/kubelet/pods/6df3bebd-0a1b-497b-bfb2-5bb85067365f/volumes"
Jul 24 17:01:30.048769 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:30.048742    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d8c754f6b-qcxmb"]
Jul 24 17:01:30.050646 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:01:30.050620    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce02e79c_6549_4c69_b469_bba26b0567cd.slice/crio-47bd7bade7886441f644c88ccaf0be0c298f8e46ef4c74fda3dc8161ce7e7288 WatchSource:0}: Error finding container 47bd7bade7886441f644c88ccaf0be0c298f8e46ef4c74fda3dc8161ce7e7288: Status 404 returned error can't find the container with id 47bd7bade7886441f644c88ccaf0be0c298f8e46ef4c74fda3dc8161ce7e7288
Jul 24 17:01:30.606010 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:30.605974    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d8c754f6b-qcxmb" event={"ID":"ce02e79c-6549-4c69-b469-bba26b0567cd","Type":"ContainerStarted","Data":"ac94afb5a6663b5d4a44a076114bae4fff6fc0f5327df8b32e58bd77093066c5"}
Jul 24 17:01:30.606010 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:30.606028    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d8c754f6b-qcxmb" event={"ID":"ce02e79c-6549-4c69-b469-bba26b0567cd","Type":"ContainerStarted","Data":"47bd7bade7886441f644c88ccaf0be0c298f8e46ef4c74fda3dc8161ce7e7288"}
Jul 24 17:01:30.606604 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:30.606115    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7d8c754f6b-qcxmb"
Jul 24 17:01:30.626237 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:30.626180    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7d8c754f6b-qcxmb" podStartSLOduration=1.2011763 podStartE2EDuration="1.626166569s" totalImagesPullingTime="424.990269ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:01:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:01:30.623247127 +0000 UTC m=+1113.265211033" watchObservedRunningTime="2026-07-24 17:01:30.626166569 +0000 UTC m=+1113.268130452"
Jul 24 17:01:35.686010 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:35.685977    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:01:35.745925 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:35.745893    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:01:41.613428 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:41.613395    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7d8c754f6b-qcxmb"
Jul 24 17:01:54.760226 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:01:54.760184    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:02:07.291538 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:02:07.291506    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:02:09.432891 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:02:09.432850    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:02:09.557004 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:02:09.556969    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:02:09.690975 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:02:09.690873    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:02:15.247158 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:02:15.247126    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:02:15.672618 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:02:15.672585    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:02:15.776579 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:02:15.776545    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:02:15.903754 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:02:15.903722    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:02:16.023998 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:02:16.023905    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:02:21.641571 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:02:21.641498    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:02:21.741894 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:02:21.741864    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:02:58.022554 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:02:58.022528    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ddklx_fb5a9838-1e05-4f86-b46c-760bd7aaa038/console-operator/1.log"
Jul 24 17:02:58.025309 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:02:58.025291    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ddklx_fb5a9838-1e05-4f86-b46c-760bd7aaa038/console-operator/1.log"
Jul 24 17:04:01.028705 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:04:01.028658    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:04:02.328559 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:04:02.328524    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:07:58.051122 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:07:58.051094    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ddklx_fb5a9838-1e05-4f86-b46c-760bd7aaa038/console-operator/1.log"
Jul 24 17:07:58.055647 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:07:58.055622    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ddklx_fb5a9838-1e05-4f86-b46c-760bd7aaa038/console-operator/1.log"
Jul 24 17:08:10.405264 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:08:10.405221    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:09:50.454080 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:09:50.454006    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:09:56.241223 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:09:56.241190    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:10:20.831324 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:10:20.831287    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:10:22.323745 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:10:22.323713    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:10:33.185043 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:10:33.184998    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:10:33.284034 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:10:33.283996    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:10:37.288106 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:10:37.288067    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:10:49.711533 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:10:49.711505    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:12:58.086184 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:12:58.086152    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ddklx_fb5a9838-1e05-4f86-b46c-760bd7aaa038/console-operator/1.log"
Jul 24 17:12:58.092346 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:12:58.092327    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ddklx_fb5a9838-1e05-4f86-b46c-760bd7aaa038/console-operator/1.log"
Jul 24 17:13:35.531137 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:13:35.531104    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:13:56.885657 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:13:56.885621    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:13:59.935100 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:13:59.935050    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:00.033455 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:00.033424    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:00.449483 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:00.449449    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:06.172829 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:06.172772    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:06.237831 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:06.237794    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:06.392211 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:06.392180    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:06.690633 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:06.690601    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:06.965002 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:06.964884    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:09.112511 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:09.112476    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:09.224036 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:09.224001    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:09.320222 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:09.320188    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0275111b-f4598fdc8-zbxg5"]
Jul 24 17:14:09.324133 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:09.324115    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0275111b-f4598fdc8-zbxg5"
Jul 24 17:14:09.326547 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:09.326525    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0275111b-serving-cert\""
Jul 24 17:14:09.333733 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:09.333558    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0275111b-f4598fdc8-zbxg5"]
Jul 24 17:14:09.352059 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:09.352031    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:09.434435 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:09.434395    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eac9cb8e-a479-45e6-bf24-12ce06f8d35e-maas-api-tls\") pod \"maas-api-e2e-ait-0275111b-f4598fdc8-zbxg5\" (UID: \"eac9cb8e-a479-45e6-bf24-12ce06f8d35e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0275111b-f4598fdc8-zbxg5"
Jul 24 17:14:09.434620 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:09.434511    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmkn9\" (UniqueName: \"kubernetes.io/projected/eac9cb8e-a479-45e6-bf24-12ce06f8d35e-kube-api-access-cmkn9\") pod \"maas-api-e2e-ait-0275111b-f4598fdc8-zbxg5\" (UID: \"eac9cb8e-a479-45e6-bf24-12ce06f8d35e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0275111b-f4598fdc8-zbxg5"
Jul 24 17:14:09.434620 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:09.434559    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eac9cb8e-a479-45e6-bf24-12ce06f8d35e-openshift-service-ca\") pod \"maas-api-e2e-ait-0275111b-f4598fdc8-zbxg5\" (UID: \"eac9cb8e-a479-45e6-bf24-12ce06f8d35e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0275111b-f4598fdc8-zbxg5"
Jul 24 17:14:09.535988 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:09.535923    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmkn9\" (UniqueName: \"kubernetes.io/projected/eac9cb8e-a479-45e6-bf24-12ce06f8d35e-kube-api-access-cmkn9\") pod \"maas-api-e2e-ait-0275111b-f4598fdc8-zbxg5\" (UID: \"eac9cb8e-a479-45e6-bf24-12ce06f8d35e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0275111b-f4598fdc8-zbxg5"
Jul 24 17:14:09.536182 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:09.536014    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eac9cb8e-a479-45e6-bf24-12ce06f8d35e-openshift-service-ca\") pod \"maas-api-e2e-ait-0275111b-f4598fdc8-zbxg5\" (UID: \"eac9cb8e-a479-45e6-bf24-12ce06f8d35e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0275111b-f4598fdc8-zbxg5"
Jul 24 17:14:09.536182 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:09.536084    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eac9cb8e-a479-45e6-bf24-12ce06f8d35e-maas-api-tls\") pod \"maas-api-e2e-ait-0275111b-f4598fdc8-zbxg5\" (UID: \"eac9cb8e-a479-45e6-bf24-12ce06f8d35e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0275111b-f4598fdc8-zbxg5"
Jul 24 17:14:09.536260 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:14:09.536181    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-0275111b-serving-cert: secret "maas-api-e2e-ait-0275111b-serving-cert" not found
Jul 24 17:14:09.536303 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:14:09.536273    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eac9cb8e-a479-45e6-bf24-12ce06f8d35e-maas-api-tls podName:eac9cb8e-a479-45e6-bf24-12ce06f8d35e nodeName:}" failed. No retries permitted until 2026-07-24 17:14:10.036250756 +0000 UTC m=+1872.678214635 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eac9cb8e-a479-45e6-bf24-12ce06f8d35e-maas-api-tls") pod "maas-api-e2e-ait-0275111b-f4598fdc8-zbxg5" (UID: "eac9cb8e-a479-45e6-bf24-12ce06f8d35e") : secret "maas-api-e2e-ait-0275111b-serving-cert" not found
Jul 24 17:14:09.536760 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:09.536741    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eac9cb8e-a479-45e6-bf24-12ce06f8d35e-openshift-service-ca\") pod \"maas-api-e2e-ait-0275111b-f4598fdc8-zbxg5\" (UID: \"eac9cb8e-a479-45e6-bf24-12ce06f8d35e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0275111b-f4598fdc8-zbxg5"
Jul 24 17:14:09.544558 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:09.544533    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmkn9\" (UniqueName: \"kubernetes.io/projected/eac9cb8e-a479-45e6-bf24-12ce06f8d35e-kube-api-access-cmkn9\") pod \"maas-api-e2e-ait-0275111b-f4598fdc8-zbxg5\" (UID: \"eac9cb8e-a479-45e6-bf24-12ce06f8d35e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0275111b-f4598fdc8-zbxg5"
Jul 24 17:14:09.768192 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:09.768111    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:09.835635 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:09.835594    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:09.931900 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:09.931866    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:09.932316 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:09.932295    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0275111b-fd6db8c96-ckljv"]
Jul 24 17:14:09.936010 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:09.935987    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0275111b-fd6db8c96-ckljv"
Jul 24 17:14:09.939162 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:09.939139    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0275111b\""
Jul 24 17:14:09.939276 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:09.939166    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0275111b-dockercfg-m6fnh\""
Jul 24 17:14:09.951915 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:09.951887    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0275111b-fd6db8c96-ckljv"]
Jul 24 17:14:09.963612 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:09.963575    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0275111b-654d654bd6-cmb44"]
Jul 24 17:14:09.966977 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:09.966955    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0275111b-654d654bd6-cmb44"
Jul 24 17:14:09.979868 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:09.979840    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0275111b-654d654bd6-cmb44"]
Jul 24 17:14:10.041020 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:10.040915    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eac9cb8e-a479-45e6-bf24-12ce06f8d35e-maas-api-tls\") pod \"maas-api-e2e-ait-0275111b-f4598fdc8-zbxg5\" (UID: \"eac9cb8e-a479-45e6-bf24-12ce06f8d35e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0275111b-f4598fdc8-zbxg5"
Jul 24 17:14:10.041020 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:10.040978    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0a22054-2415-4ea7-a102-426d8319b085-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0275111b-fd6db8c96-ckljv\" (UID: \"e0a22054-2415-4ea7-a102-426d8319b085\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0275111b-fd6db8c96-ckljv"
Jul 24 17:14:10.041256 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:10.041044    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd2f2\" (UniqueName: \"kubernetes.io/projected/e0a22054-2415-4ea7-a102-426d8319b085-kube-api-access-dd2f2\") pod \"payload-pre-processing-e2e-ait-0275111b-fd6db8c96-ckljv\" (UID: \"e0a22054-2415-4ea7-a102-426d8319b085\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0275111b-fd6db8c96-ckljv"
Jul 24 17:14:10.044097 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:10.044067    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eac9cb8e-a479-45e6-bf24-12ce06f8d35e-maas-api-tls\") pod \"maas-api-e2e-ait-0275111b-f4598fdc8-zbxg5\" (UID: \"eac9cb8e-a479-45e6-bf24-12ce06f8d35e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0275111b-f4598fdc8-zbxg5"
Jul 24 17:14:10.141518 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:10.141472    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd2f2\" (UniqueName: \"kubernetes.io/projected/e0a22054-2415-4ea7-a102-426d8319b085-kube-api-access-dd2f2\") pod \"payload-pre-processing-e2e-ait-0275111b-fd6db8c96-ckljv\" (UID: \"e0a22054-2415-4ea7-a102-426d8319b085\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0275111b-fd6db8c96-ckljv"
Jul 24 17:14:10.141977 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:10.141563    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99zk6\" (UniqueName: \"kubernetes.io/projected/82d6826b-bb29-4b9c-8488-ddf0d76a3334-kube-api-access-99zk6\") pod \"payload-processing-e2e-ait-0275111b-654d654bd6-cmb44\" (UID: \"82d6826b-bb29-4b9c-8488-ddf0d76a3334\") " pod="openshift-ingress/payload-processing-e2e-ait-0275111b-654d654bd6-cmb44"
Jul 24 17:14:10.141977 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:10.141587    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0a22054-2415-4ea7-a102-426d8319b085-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0275111b-fd6db8c96-ckljv\" (UID: \"e0a22054-2415-4ea7-a102-426d8319b085\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0275111b-fd6db8c96-ckljv"
Jul 24 17:14:10.141977 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:10.141604    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d6826b-bb29-4b9c-8488-ddf0d76a3334-plugins-config-volume\") pod \"payload-processing-e2e-ait-0275111b-654d654bd6-cmb44\" (UID: \"82d6826b-bb29-4b9c-8488-ddf0d76a3334\") " pod="openshift-ingress/payload-processing-e2e-ait-0275111b-654d654bd6-cmb44"
Jul 24 17:14:10.142474 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:10.142450    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0a22054-2415-4ea7-a102-426d8319b085-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0275111b-fd6db8c96-ckljv\" (UID: \"e0a22054-2415-4ea7-a102-426d8319b085\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0275111b-fd6db8c96-ckljv"
Jul 24 17:14:10.150992 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:10.150965    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd2f2\" (UniqueName: \"kubernetes.io/projected/e0a22054-2415-4ea7-a102-426d8319b085-kube-api-access-dd2f2\") pod \"payload-pre-processing-e2e-ait-0275111b-fd6db8c96-ckljv\" (UID: \"e0a22054-2415-4ea7-a102-426d8319b085\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0275111b-fd6db8c96-ckljv"
Jul 24 17:14:10.238164 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:10.238127    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0275111b-f4598fdc8-zbxg5"
Jul 24 17:14:10.242536 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:10.242509    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99zk6\" (UniqueName: \"kubernetes.io/projected/82d6826b-bb29-4b9c-8488-ddf0d76a3334-kube-api-access-99zk6\") pod \"payload-processing-e2e-ait-0275111b-654d654bd6-cmb44\" (UID: \"82d6826b-bb29-4b9c-8488-ddf0d76a3334\") " pod="openshift-ingress/payload-processing-e2e-ait-0275111b-654d654bd6-cmb44"
Jul 24 17:14:10.242622 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:10.242549    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d6826b-bb29-4b9c-8488-ddf0d76a3334-plugins-config-volume\") pod \"payload-processing-e2e-ait-0275111b-654d654bd6-cmb44\" (UID: \"82d6826b-bb29-4b9c-8488-ddf0d76a3334\") " pod="openshift-ingress/payload-processing-e2e-ait-0275111b-654d654bd6-cmb44"
Jul 24 17:14:10.243218 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:10.243199    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d6826b-bb29-4b9c-8488-ddf0d76a3334-plugins-config-volume\") pod \"payload-processing-e2e-ait-0275111b-654d654bd6-cmb44\" (UID: \"82d6826b-bb29-4b9c-8488-ddf0d76a3334\") " pod="openshift-ingress/payload-processing-e2e-ait-0275111b-654d654bd6-cmb44"
Jul 24 17:14:10.247056 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:10.247032    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0275111b-fd6db8c96-ckljv"
Jul 24 17:14:10.250757 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:10.250732    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99zk6\" (UniqueName: \"kubernetes.io/projected/82d6826b-bb29-4b9c-8488-ddf0d76a3334-kube-api-access-99zk6\") pod \"payload-processing-e2e-ait-0275111b-654d654bd6-cmb44\" (UID: \"82d6826b-bb29-4b9c-8488-ddf0d76a3334\") " pod="openshift-ingress/payload-processing-e2e-ait-0275111b-654d654bd6-cmb44"
Jul 24 17:14:10.280267 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:10.280220    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0275111b-654d654bd6-cmb44"
Jul 24 17:14:10.405643 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:10.405618    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0275111b-f4598fdc8-zbxg5"]
Jul 24 17:14:10.408315 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:14:10.408283    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeac9cb8e_a479_45e6_bf24_12ce06f8d35e.slice/crio-c3ff318f4cceea8a6d89389a2d023c6af38414ffeeb70701acf6e0e8d41b7341 WatchSource:0}: Error finding container c3ff318f4cceea8a6d89389a2d023c6af38414ffeeb70701acf6e0e8d41b7341: Status 404 returned error can't find the container with id c3ff318f4cceea8a6d89389a2d023c6af38414ffeeb70701acf6e0e8d41b7341
Jul 24 17:14:10.410540 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:10.410520    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 17:14:10.467003 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:10.466979    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0275111b-654d654bd6-cmb44"]
Jul 24 17:14:10.469128 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:14:10.469100    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82d6826b_bb29_4b9c_8488_ddf0d76a3334.slice/crio-4d59356e9425ce49eb13cb8a816bc827fc7a7369371382bbdb80577833f29964 WatchSource:0}: Error finding container 4d59356e9425ce49eb13cb8a816bc827fc7a7369371382bbdb80577833f29964: Status 404 returned error can't find the container with id 4d59356e9425ce49eb13cb8a816bc827fc7a7369371382bbdb80577833f29964
Jul 24 17:14:10.643599 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:10.643574    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0275111b-fd6db8c96-ckljv"]
Jul 24 17:14:10.645885 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:14:10.645859    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a22054_2415_4ea7_a102_426d8319b085.slice/crio-93f6c0d8b91f452f4dcb93c357f4d8b43047627babf90b53edd777f714935504 WatchSource:0}: Error finding container 93f6c0d8b91f452f4dcb93c357f4d8b43047627babf90b53edd777f714935504: Status 404 returned error can't find the container with id 93f6c0d8b91f452f4dcb93c357f4d8b43047627babf90b53edd777f714935504
Jul 24 17:14:11.210557 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:11.210524    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0275111b-f4598fdc8-zbxg5" event={"ID":"eac9cb8e-a479-45e6-bf24-12ce06f8d35e","Type":"ContainerStarted","Data":"5af39f892d72da26b6d14615c197d3ca584c149e905f02ffc3178c0655a6a25c"}
Jul 24 17:14:11.211052 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:11.210583    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0275111b-f4598fdc8-zbxg5" event={"ID":"eac9cb8e-a479-45e6-bf24-12ce06f8d35e","Type":"ContainerStarted","Data":"c3ff318f4cceea8a6d89389a2d023c6af38414ffeeb70701acf6e0e8d41b7341"}
Jul 24 17:14:11.211052 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:11.210694    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0275111b-f4598fdc8-zbxg5"
Jul 24 17:14:11.212021 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:11.211996    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0275111b-654d654bd6-cmb44" event={"ID":"82d6826b-bb29-4b9c-8488-ddf0d76a3334","Type":"ContainerStarted","Data":"112a4a575b5ec4fd649afa84b9e9c5e9935f84459da697e16dba33f73ca66b39"}
Jul 24 17:14:11.212133 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:11.212055    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0275111b-654d654bd6-cmb44"
Jul 24 17:14:11.212133 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:11.212079    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0275111b-654d654bd6-cmb44" event={"ID":"82d6826b-bb29-4b9c-8488-ddf0d76a3334","Type":"ContainerStarted","Data":"4d59356e9425ce49eb13cb8a816bc827fc7a7369371382bbdb80577833f29964"}
Jul 24 17:14:11.213500 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:11.213477    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0275111b-fd6db8c96-ckljv" event={"ID":"e0a22054-2415-4ea7-a102-426d8319b085","Type":"ContainerStarted","Data":"6681c61b19dd64d3610eb7b1d08f41a9db80d1644d7c92e9998bb92655cbfad2"}
Jul 24 17:14:11.213623 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:11.213513    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0275111b-fd6db8c96-ckljv" event={"ID":"e0a22054-2415-4ea7-a102-426d8319b085","Type":"ContainerStarted","Data":"93f6c0d8b91f452f4dcb93c357f4d8b43047627babf90b53edd777f714935504"}
Jul 24 17:14:11.213623 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:11.213597    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0275111b-fd6db8c96-ckljv"
Jul 24 17:14:11.227878 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:11.227828    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0275111b-f4598fdc8-zbxg5" podStartSLOduration=1.760778671 podStartE2EDuration="2.227808824s" totalImagesPullingTime="467.030153ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:14:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:14:11.225370304 +0000 UTC m=+1873.867334188" watchObservedRunningTime="2026-07-24 17:14:11.227808824 +0000 UTC m=+1873.869772709"
Jul 24 17:14:11.241308 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:11.241259    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0275111b-fd6db8c96-ckljv" podStartSLOduration=2.241242321 podStartE2EDuration="2.241242321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:14:11.239627986 +0000 UTC m=+1873.881591907" watchObservedRunningTime="2026-07-24 17:14:11.241242321 +0000 UTC m=+1873.883206205"
Jul 24 17:14:11.258689 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:11.258635    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0275111b-654d654bd6-cmb44" podStartSLOduration=2.25862021 podStartE2EDuration="2.25862021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:14:11.256959156 +0000 UTC m=+1873.898923036" watchObservedRunningTime="2026-07-24 17:14:11.25862021 +0000 UTC m=+1873.900584093"
Jul 24 17:14:17.224201 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:17.224170    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0275111b-f4598fdc8-zbxg5"
Jul 24 17:14:22.220436 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:22.220352    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0275111b-fd6db8c96-ckljv"
Jul 24 17:14:22.220880 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:22.220545    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0275111b-654d654bd6-cmb44"
Jul 24 17:14:45.142578 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:45.142547    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0275111b-f4598fdc8-zbxg5"]
Jul 24 17:14:45.143190 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:45.142792    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0275111b-f4598fdc8-zbxg5" podUID="eac9cb8e-a479-45e6-bf24-12ce06f8d35e" containerName="maas-api" containerID="cri-o://5af39f892d72da26b6d14615c197d3ca584c149e905f02ffc3178c0655a6a25c" gracePeriod=30
Jul 24 17:14:45.177048 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:45.177021    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:45.210627 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:45.210595    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0275111b-654d654bd6-cmb44"]
Jul 24 17:14:45.210967 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:45.210900    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0275111b-654d654bd6-cmb44" podUID="82d6826b-bb29-4b9c-8488-ddf0d76a3334" containerName="payload-processing" containerID="cri-o://112a4a575b5ec4fd649afa84b9e9c5e9935f84459da697e16dba33f73ca66b39" gracePeriod=30
Jul 24 17:14:45.215961 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:45.215543    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0275111b-fd6db8c96-ckljv"]
Jul 24 17:14:45.215961 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:45.215753    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0275111b-fd6db8c96-ckljv" podUID="e0a22054-2415-4ea7-a102-426d8319b085" containerName="payload-pre-processing" containerID="cri-o://6681c61b19dd64d3610eb7b1d08f41a9db80d1644d7c92e9998bb92655cbfad2" gracePeriod=30
Jul 24 17:14:45.251040 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:45.251007    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:45.315756 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:45.315730    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:45.390072 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:45.389691    2563 generic.go:356] "Generic (PLEG): container finished" podID="e0a22054-2415-4ea7-a102-426d8319b085" containerID="6681c61b19dd64d3610eb7b1d08f41a9db80d1644d7c92e9998bb92655cbfad2" exitCode=0
Jul 24 17:14:45.390072 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:45.389836    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0275111b-fd6db8c96-ckljv" event={"ID":"e0a22054-2415-4ea7-a102-426d8319b085","Type":"ContainerDied","Data":"6681c61b19dd64d3610eb7b1d08f41a9db80d1644d7c92e9998bb92655cbfad2"}
Jul 24 17:14:45.391856 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:45.391644    2563 generic.go:356] "Generic (PLEG): container finished" podID="eac9cb8e-a479-45e6-bf24-12ce06f8d35e" containerID="5af39f892d72da26b6d14615c197d3ca584c149e905f02ffc3178c0655a6a25c" exitCode=0
Jul 24 17:14:45.391856 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:45.391791    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0275111b-f4598fdc8-zbxg5" event={"ID":"eac9cb8e-a479-45e6-bf24-12ce06f8d35e","Type":"ContainerDied","Data":"5af39f892d72da26b6d14615c197d3ca584c149e905f02ffc3178c0655a6a25c"}
Jul 24 17:14:45.394155 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:45.394024    2563 generic.go:356] "Generic (PLEG): container finished" podID="82d6826b-bb29-4b9c-8488-ddf0d76a3334" containerID="112a4a575b5ec4fd649afa84b9e9c5e9935f84459da697e16dba33f73ca66b39" exitCode=0
Jul 24 17:14:45.394155 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:45.394059    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0275111b-654d654bd6-cmb44" event={"ID":"82d6826b-bb29-4b9c-8488-ddf0d76a3334","Type":"ContainerDied","Data":"112a4a575b5ec4fd649afa84b9e9c5e9935f84459da697e16dba33f73ca66b39"}
Jul 24 17:14:45.477361 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:45.477334    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0275111b-f4598fdc8-zbxg5"
Jul 24 17:14:45.481602 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:45.481579    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0275111b-654d654bd6-cmb44"
Jul 24 17:14:45.509318 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:45.509286    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0275111b-fd6db8c96-ckljv"
Jul 24 17:14:45.556219 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:45.556185    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eac9cb8e-a479-45e6-bf24-12ce06f8d35e-openshift-service-ca\") pod \"eac9cb8e-a479-45e6-bf24-12ce06f8d35e\" (UID: \"eac9cb8e-a479-45e6-bf24-12ce06f8d35e\") "
Jul 24 17:14:45.556386 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:45.556268    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99zk6\" (UniqueName: \"kubernetes.io/projected/82d6826b-bb29-4b9c-8488-ddf0d76a3334-kube-api-access-99zk6\") pod \"82d6826b-bb29-4b9c-8488-ddf0d76a3334\" (UID: \"82d6826b-bb29-4b9c-8488-ddf0d76a3334\") "
Jul 24 17:14:45.556386 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:45.556310    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0a22054-2415-4ea7-a102-426d8319b085-plugins-config-volume\") pod \"e0a22054-2415-4ea7-a102-426d8319b085\" (UID: \"e0a22054-2415-4ea7-a102-426d8319b085\") "
Jul 24 17:14:45.556386 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:45.556339    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d6826b-bb29-4b9c-8488-ddf0d76a3334-plugins-config-volume\") pod \"82d6826b-bb29-4b9c-8488-ddf0d76a3334\" (UID: \"82d6826b-bb29-4b9c-8488-ddf0d76a3334\") "
Jul 24 17:14:45.556386 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:45.556368    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dd2f2\" (UniqueName: \"kubernetes.io/projected/e0a22054-2415-4ea7-a102-426d8319b085-kube-api-access-dd2f2\") pod \"e0a22054-2415-4ea7-a102-426d8319b085\" (UID: \"e0a22054-2415-4ea7-a102-426d8319b085\") "
Jul 24 17:14:45.556598 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:45.556396    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eac9cb8e-a479-45e6-bf24-12ce06f8d35e-maas-api-tls\") pod \"eac9cb8e-a479-45e6-bf24-12ce06f8d35e\" (UID: \"eac9cb8e-a479-45e6-bf24-12ce06f8d35e\") "
Jul 24 17:14:45.556598 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:45.556435    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmkn9\" (UniqueName: \"kubernetes.io/projected/eac9cb8e-a479-45e6-bf24-12ce06f8d35e-kube-api-access-cmkn9\") pod \"eac9cb8e-a479-45e6-bf24-12ce06f8d35e\" (UID: \"eac9cb8e-a479-45e6-bf24-12ce06f8d35e\") "
Jul 24 17:14:45.556700 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:45.556587    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eac9cb8e-a479-45e6-bf24-12ce06f8d35e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eac9cb8e-a479-45e6-bf24-12ce06f8d35e" (UID: "eac9cb8e-a479-45e6-bf24-12ce06f8d35e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:14:45.556754 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:45.556705    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0a22054-2415-4ea7-a102-426d8319b085-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e0a22054-2415-4ea7-a102-426d8319b085" (UID: "e0a22054-2415-4ea7-a102-426d8319b085"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:14:45.556754 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:45.556713    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82d6826b-bb29-4b9c-8488-ddf0d76a3334-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "82d6826b-bb29-4b9c-8488-ddf0d76a3334" (UID: "82d6826b-bb29-4b9c-8488-ddf0d76a3334"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:14:45.556868 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:45.556795    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0a22054-2415-4ea7-a102-426d8319b085-plugins-config-volume\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:14:45.556868 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:45.556811    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d6826b-bb29-4b9c-8488-ddf0d76a3334-plugins-config-volume\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:14:45.556868 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:45.556826    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eac9cb8e-a479-45e6-bf24-12ce06f8d35e-openshift-service-ca\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:14:45.558555 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:45.558533    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eac9cb8e-a479-45e6-bf24-12ce06f8d35e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eac9cb8e-a479-45e6-bf24-12ce06f8d35e" (UID: "eac9cb8e-a479-45e6-bf24-12ce06f8d35e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:14:45.558649 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:45.558618    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eac9cb8e-a479-45e6-bf24-12ce06f8d35e-kube-api-access-cmkn9" (OuterVolumeSpecName: "kube-api-access-cmkn9") pod "eac9cb8e-a479-45e6-bf24-12ce06f8d35e" (UID: "eac9cb8e-a479-45e6-bf24-12ce06f8d35e"). InnerVolumeSpecName "kube-api-access-cmkn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:14:45.558992 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:45.558971    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82d6826b-bb29-4b9c-8488-ddf0d76a3334-kube-api-access-99zk6" (OuterVolumeSpecName: "kube-api-access-99zk6") pod "82d6826b-bb29-4b9c-8488-ddf0d76a3334" (UID: "82d6826b-bb29-4b9c-8488-ddf0d76a3334"). InnerVolumeSpecName "kube-api-access-99zk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:14:45.559078 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:45.558997    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0a22054-2415-4ea7-a102-426d8319b085-kube-api-access-dd2f2" (OuterVolumeSpecName: "kube-api-access-dd2f2") pod "e0a22054-2415-4ea7-a102-426d8319b085" (UID: "e0a22054-2415-4ea7-a102-426d8319b085"). InnerVolumeSpecName "kube-api-access-dd2f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:14:45.663972 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:45.663863    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99zk6\" (UniqueName: \"kubernetes.io/projected/82d6826b-bb29-4b9c-8488-ddf0d76a3334-kube-api-access-99zk6\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:14:45.663972 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:45.663897    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd2f2\" (UniqueName: \"kubernetes.io/projected/e0a22054-2415-4ea7-a102-426d8319b085-kube-api-access-dd2f2\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:14:45.663972 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:45.663914    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eac9cb8e-a479-45e6-bf24-12ce06f8d35e-maas-api-tls\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:14:45.663972 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:45.663927    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmkn9\" (UniqueName: \"kubernetes.io/projected/eac9cb8e-a479-45e6-bf24-12ce06f8d35e-kube-api-access-cmkn9\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:14:46.399657 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:46.399627    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0275111b-fd6db8c96-ckljv"
Jul 24 17:14:46.400118 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:46.399631    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0275111b-fd6db8c96-ckljv" event={"ID":"e0a22054-2415-4ea7-a102-426d8319b085","Type":"ContainerDied","Data":"93f6c0d8b91f452f4dcb93c357f4d8b43047627babf90b53edd777f714935504"}
Jul 24 17:14:46.400118 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:46.399770    2563 scope.go:117] "RemoveContainer" containerID="6681c61b19dd64d3610eb7b1d08f41a9db80d1644d7c92e9998bb92655cbfad2"
Jul 24 17:14:46.401183 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:46.400954    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0275111b-f4598fdc8-zbxg5"
Jul 24 17:14:46.401183 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:46.400984    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0275111b-f4598fdc8-zbxg5" event={"ID":"eac9cb8e-a479-45e6-bf24-12ce06f8d35e","Type":"ContainerDied","Data":"c3ff318f4cceea8a6d89389a2d023c6af38414ffeeb70701acf6e0e8d41b7341"}
Jul 24 17:14:46.402387 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:46.402358    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0275111b-654d654bd6-cmb44" event={"ID":"82d6826b-bb29-4b9c-8488-ddf0d76a3334","Type":"ContainerDied","Data":"4d59356e9425ce49eb13cb8a816bc827fc7a7369371382bbdb80577833f29964"}
Jul 24 17:14:46.402387 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:46.402383    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0275111b-654d654bd6-cmb44"
Jul 24 17:14:46.409211 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:46.409193    2563 scope.go:117] "RemoveContainer" containerID="5af39f892d72da26b6d14615c197d3ca584c149e905f02ffc3178c0655a6a25c"
Jul 24 17:14:46.416790 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:46.416773    2563 scope.go:117] "RemoveContainer" containerID="112a4a575b5ec4fd649afa84b9e9c5e9935f84459da697e16dba33f73ca66b39"
Jul 24 17:14:46.421694 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:46.421671    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0275111b-fd6db8c96-ckljv"]
Jul 24 17:14:46.425243 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:46.425223    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0275111b-fd6db8c96-ckljv"]
Jul 24 17:14:46.435029 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:46.435010    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0275111b-654d654bd6-cmb44"]
Jul 24 17:14:46.439916 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:46.439897    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0275111b-654d654bd6-cmb44"]
Jul 24 17:14:46.449319 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:46.449299    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0275111b-f4598fdc8-zbxg5"]
Jul 24 17:14:46.452799 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:46.452780    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0275111b-f4598fdc8-zbxg5"]
Jul 24 17:14:48.002796 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:48.002762    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82d6826b-bb29-4b9c-8488-ddf0d76a3334" path="/var/lib/kubelet/pods/82d6826b-bb29-4b9c-8488-ddf0d76a3334/volumes"
Jul 24 17:14:48.003185 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:48.003134    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0a22054-2415-4ea7-a102-426d8319b085" path="/var/lib/kubelet/pods/e0a22054-2415-4ea7-a102-426d8319b085/volumes"
Jul 24 17:14:48.003477 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:48.003463    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eac9cb8e-a479-45e6-bf24-12ce06f8d35e" path="/var/lib/kubelet/pods/eac9cb8e-a479-45e6-bf24-12ce06f8d35e/volumes"
Jul 24 17:14:49.683191 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:49.683144    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:49.840785 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:49.840753    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:50.075380 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:50.075302    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:50.308301 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:50.308269    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:50.408468 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:50.408440    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:50.532583 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:50.532405    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:50.867798 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:50.867713    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:51.693180 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:51.693149    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:53.454024 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:53.453988    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:53.533519 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:53.533486    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9651e0f7-6b7fb7768c-vrwgv"]
Jul 24 17:14:53.534076 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:53.534059    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82d6826b-bb29-4b9c-8488-ddf0d76a3334" containerName="payload-processing"
Jul 24 17:14:53.534125 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:53.534079    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="82d6826b-bb29-4b9c-8488-ddf0d76a3334" containerName="payload-processing"
Jul 24 17:14:53.534125 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:53.534099    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0a22054-2415-4ea7-a102-426d8319b085" containerName="payload-pre-processing"
Jul 24 17:14:53.534125 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:53.534108    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0a22054-2415-4ea7-a102-426d8319b085" containerName="payload-pre-processing"
Jul 24 17:14:53.534219 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:53.534125    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eac9cb8e-a479-45e6-bf24-12ce06f8d35e" containerName="maas-api"
Jul 24 17:14:53.534219 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:53.534133    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="eac9cb8e-a479-45e6-bf24-12ce06f8d35e" containerName="maas-api"
Jul 24 17:14:53.534219 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:53.534215    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="82d6826b-bb29-4b9c-8488-ddf0d76a3334" containerName="payload-processing"
Jul 24 17:14:53.534312 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:53.534227    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="eac9cb8e-a479-45e6-bf24-12ce06f8d35e" containerName="maas-api"
Jul 24 17:14:53.534312 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:53.534239    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0a22054-2415-4ea7-a102-426d8319b085" containerName="payload-pre-processing"
Jul 24 17:14:53.537510 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:53.537490    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9651e0f7-6b7fb7768c-vrwgv"
Jul 24 17:14:53.539741 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:53.539720    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-9651e0f7-serving-cert\""
Jul 24 17:14:53.545715 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:53.545692    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9651e0f7-6b7fb7768c-vrwgv"]
Jul 24 17:14:53.635280 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:53.635249    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1284452-e388-432a-95d7-85bf333a0176-openshift-service-ca\") pod \"maas-api-e2e-migrate-9651e0f7-6b7fb7768c-vrwgv\" (UID: \"e1284452-e388-432a-95d7-85bf333a0176\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9651e0f7-6b7fb7768c-vrwgv"
Jul 24 17:14:53.635455 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:53.635367    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9qjz\" (UniqueName: \"kubernetes.io/projected/e1284452-e388-432a-95d7-85bf333a0176-kube-api-access-v9qjz\") pod \"maas-api-e2e-migrate-9651e0f7-6b7fb7768c-vrwgv\" (UID: \"e1284452-e388-432a-95d7-85bf333a0176\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9651e0f7-6b7fb7768c-vrwgv"
Jul 24 17:14:53.635455 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:53.635408    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1284452-e388-432a-95d7-85bf333a0176-maas-api-tls\") pod \"maas-api-e2e-migrate-9651e0f7-6b7fb7768c-vrwgv\" (UID: \"e1284452-e388-432a-95d7-85bf333a0176\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9651e0f7-6b7fb7768c-vrwgv"
Jul 24 17:14:53.736490 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:53.736395    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9qjz\" (UniqueName: \"kubernetes.io/projected/e1284452-e388-432a-95d7-85bf333a0176-kube-api-access-v9qjz\") pod \"maas-api-e2e-migrate-9651e0f7-6b7fb7768c-vrwgv\" (UID: \"e1284452-e388-432a-95d7-85bf333a0176\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9651e0f7-6b7fb7768c-vrwgv"
Jul 24 17:14:53.736490 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:53.736457    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1284452-e388-432a-95d7-85bf333a0176-maas-api-tls\") pod \"maas-api-e2e-migrate-9651e0f7-6b7fb7768c-vrwgv\" (UID: \"e1284452-e388-432a-95d7-85bf333a0176\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9651e0f7-6b7fb7768c-vrwgv"
Jul 24 17:14:53.736490 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:53.736489    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1284452-e388-432a-95d7-85bf333a0176-openshift-service-ca\") pod \"maas-api-e2e-migrate-9651e0f7-6b7fb7768c-vrwgv\" (UID: \"e1284452-e388-432a-95d7-85bf333a0176\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9651e0f7-6b7fb7768c-vrwgv"
Jul 24 17:14:53.736783 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:14:53.736616    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-9651e0f7-serving-cert: secret "maas-api-e2e-migrate-9651e0f7-serving-cert" not found
Jul 24 17:14:53.736783 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:14:53.736696    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1284452-e388-432a-95d7-85bf333a0176-maas-api-tls podName:e1284452-e388-432a-95d7-85bf333a0176 nodeName:}" failed. No retries permitted until 2026-07-24 17:14:54.236678836 +0000 UTC m=+1916.878642700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e1284452-e388-432a-95d7-85bf333a0176-maas-api-tls") pod "maas-api-e2e-migrate-9651e0f7-6b7fb7768c-vrwgv" (UID: "e1284452-e388-432a-95d7-85bf333a0176") : secret "maas-api-e2e-migrate-9651e0f7-serving-cert" not found
Jul 24 17:14:53.737232 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:53.737209    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1284452-e388-432a-95d7-85bf333a0176-openshift-service-ca\") pod \"maas-api-e2e-migrate-9651e0f7-6b7fb7768c-vrwgv\" (UID: \"e1284452-e388-432a-95d7-85bf333a0176\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9651e0f7-6b7fb7768c-vrwgv"
Jul 24 17:14:53.745819 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:53.745797    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9qjz\" (UniqueName: \"kubernetes.io/projected/e1284452-e388-432a-95d7-85bf333a0176-kube-api-access-v9qjz\") pod \"maas-api-e2e-migrate-9651e0f7-6b7fb7768c-vrwgv\" (UID: \"e1284452-e388-432a-95d7-85bf333a0176\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9651e0f7-6b7fb7768c-vrwgv"
Jul 24 17:14:54.175602 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:54.175569    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9651e0f7-65d5b9b5cb-t77ph"]
Jul 24 17:14:54.179102 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:54.179086    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9651e0f7-65d5b9b5cb-t77ph"
Jul 24 17:14:54.183860 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:54.183839    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-9651e0f7-dockercfg-pll8q\""
Jul 24 17:14:54.183992 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:54.183926    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-9651e0f7\""
Jul 24 17:14:54.191543 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:54.191519    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9651e0f7-65d5b9b5cb-t77ph"]
Jul 24 17:14:54.207312 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:54.207283    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9651e0f7-6bb6685777-sdxnw"]
Jul 24 17:14:54.210826 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:54.210808    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9651e0f7-6bb6685777-sdxnw"
Jul 24 17:14:54.222025 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:54.222002    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9651e0f7-6bb6685777-sdxnw"]
Jul 24 17:14:54.241255 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:54.241224    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1284452-e388-432a-95d7-85bf333a0176-maas-api-tls\") pod \"maas-api-e2e-migrate-9651e0f7-6b7fb7768c-vrwgv\" (UID: \"e1284452-e388-432a-95d7-85bf333a0176\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9651e0f7-6b7fb7768c-vrwgv"
Jul 24 17:14:54.243659 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:54.243634    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1284452-e388-432a-95d7-85bf333a0176-maas-api-tls\") pod \"maas-api-e2e-migrate-9651e0f7-6b7fb7768c-vrwgv\" (UID: \"e1284452-e388-432a-95d7-85bf333a0176\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9651e0f7-6b7fb7768c-vrwgv"
Jul 24 17:14:54.341859 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:54.341818    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrfph\" (UniqueName: \"kubernetes.io/projected/cf87e19d-94b7-4394-bd82-3aace42bc0e8-kube-api-access-rrfph\") pod \"payload-pre-processing-e2e-migrate-9651e0f7-65d5b9b5cb-t77ph\" (UID: \"cf87e19d-94b7-4394-bd82-3aace42bc0e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9651e0f7-65d5b9b5cb-t77ph"
Jul 24 17:14:54.341859 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:54.341865    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf87e19d-94b7-4394-bd82-3aace42bc0e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-9651e0f7-65d5b9b5cb-t77ph\" (UID: \"cf87e19d-94b7-4394-bd82-3aace42bc0e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9651e0f7-65d5b9b5cb-t77ph"
Jul 24 17:14:54.342090 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:54.341886    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shqz8\" (UniqueName: \"kubernetes.io/projected/0f41db40-8e20-4034-8b3d-7ee308d5c277-kube-api-access-shqz8\") pod \"payload-processing-e2e-migrate-9651e0f7-6bb6685777-sdxnw\" (UID: \"0f41db40-8e20-4034-8b3d-7ee308d5c277\") " pod="openshift-ingress/payload-processing-e2e-migrate-9651e0f7-6bb6685777-sdxnw"
Jul 24 17:14:54.342090 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:54.341931    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f41db40-8e20-4034-8b3d-7ee308d5c277-plugins-config-volume\") pod \"payload-processing-e2e-migrate-9651e0f7-6bb6685777-sdxnw\" (UID: \"0f41db40-8e20-4034-8b3d-7ee308d5c277\") " pod="openshift-ingress/payload-processing-e2e-migrate-9651e0f7-6bb6685777-sdxnw"
Jul 24 17:14:54.442463 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:54.442379    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrfph\" (UniqueName: \"kubernetes.io/projected/cf87e19d-94b7-4394-bd82-3aace42bc0e8-kube-api-access-rrfph\") pod \"payload-pre-processing-e2e-migrate-9651e0f7-65d5b9b5cb-t77ph\" (UID: \"cf87e19d-94b7-4394-bd82-3aace42bc0e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9651e0f7-65d5b9b5cb-t77ph"
Jul 24 17:14:54.442463 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:54.442412    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf87e19d-94b7-4394-bd82-3aace42bc0e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-9651e0f7-65d5b9b5cb-t77ph\" (UID: \"cf87e19d-94b7-4394-bd82-3aace42bc0e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9651e0f7-65d5b9b5cb-t77ph"
Jul 24 17:14:54.442463 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:54.442441    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shqz8\" (UniqueName: \"kubernetes.io/projected/0f41db40-8e20-4034-8b3d-7ee308d5c277-kube-api-access-shqz8\") pod \"payload-processing-e2e-migrate-9651e0f7-6bb6685777-sdxnw\" (UID: \"0f41db40-8e20-4034-8b3d-7ee308d5c277\") " pod="openshift-ingress/payload-processing-e2e-migrate-9651e0f7-6bb6685777-sdxnw"
Jul 24 17:14:54.442705 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:54.442475    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f41db40-8e20-4034-8b3d-7ee308d5c277-plugins-config-volume\") pod \"payload-processing-e2e-migrate-9651e0f7-6bb6685777-sdxnw\" (UID: \"0f41db40-8e20-4034-8b3d-7ee308d5c277\") " pod="openshift-ingress/payload-processing-e2e-migrate-9651e0f7-6bb6685777-sdxnw"
Jul 24 17:14:54.443254 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:54.443232    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f41db40-8e20-4034-8b3d-7ee308d5c277-plugins-config-volume\") pod \"payload-processing-e2e-migrate-9651e0f7-6bb6685777-sdxnw\" (UID: \"0f41db40-8e20-4034-8b3d-7ee308d5c277\") " pod="openshift-ingress/payload-processing-e2e-migrate-9651e0f7-6bb6685777-sdxnw"
Jul 24 17:14:54.443360 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:54.443232    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf87e19d-94b7-4394-bd82-3aace42bc0e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-9651e0f7-65d5b9b5cb-t77ph\" (UID: \"cf87e19d-94b7-4394-bd82-3aace42bc0e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9651e0f7-65d5b9b5cb-t77ph"
Jul 24 17:14:54.450273 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:54.450245    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9651e0f7-6b7fb7768c-vrwgv"
Jul 24 17:14:54.450391 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:54.450321    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrfph\" (UniqueName: \"kubernetes.io/projected/cf87e19d-94b7-4394-bd82-3aace42bc0e8-kube-api-access-rrfph\") pod \"payload-pre-processing-e2e-migrate-9651e0f7-65d5b9b5cb-t77ph\" (UID: \"cf87e19d-94b7-4394-bd82-3aace42bc0e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9651e0f7-65d5b9b5cb-t77ph"
Jul 24 17:14:54.450628 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:54.450611    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shqz8\" (UniqueName: \"kubernetes.io/projected/0f41db40-8e20-4034-8b3d-7ee308d5c277-kube-api-access-shqz8\") pod \"payload-processing-e2e-migrate-9651e0f7-6bb6685777-sdxnw\" (UID: \"0f41db40-8e20-4034-8b3d-7ee308d5c277\") " pod="openshift-ingress/payload-processing-e2e-migrate-9651e0f7-6bb6685777-sdxnw"
Jul 24 17:14:54.490591 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:54.490556    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9651e0f7-65d5b9b5cb-t77ph"
Jul 24 17:14:54.521906 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:54.521854    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9651e0f7-6bb6685777-sdxnw"
Jul 24 17:14:54.611645 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:54.611550    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9651e0f7-6b7fb7768c-vrwgv"]
Jul 24 17:14:54.614848 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:14:54.614814    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1284452_e388_432a_95d7_85bf333a0176.slice/crio-b0d89a44dc1d1fe5c9a82a1b4d69fc6a42083090d3f60e393a08b614e0968365 WatchSource:0}: Error finding container b0d89a44dc1d1fe5c9a82a1b4d69fc6a42083090d3f60e393a08b614e0968365: Status 404 returned error can't find the container with id b0d89a44dc1d1fe5c9a82a1b4d69fc6a42083090d3f60e393a08b614e0968365
Jul 24 17:14:54.680119 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:54.680087    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9651e0f7-6bb6685777-sdxnw"]
Jul 24 17:14:54.680995 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:14:54.680965    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f41db40_8e20_4034_8b3d_7ee308d5c277.slice/crio-a9b9a35af0f960f8e288aaff2227d80c09e4a4789756e8f17f26fec3cd0901a0 WatchSource:0}: Error finding container a9b9a35af0f960f8e288aaff2227d80c09e4a4789756e8f17f26fec3cd0901a0: Status 404 returned error can't find the container with id a9b9a35af0f960f8e288aaff2227d80c09e4a4789756e8f17f26fec3cd0901a0
Jul 24 17:14:54.854762 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:54.854731    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9651e0f7-65d5b9b5cb-t77ph"]
Jul 24 17:14:54.857166 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:14:54.857133    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf87e19d_94b7_4394_bd82_3aace42bc0e8.slice/crio-ccf1242dc798e4da2cdfac4022669f3544dad2db0b3a024ab23669816258bf0b WatchSource:0}: Error finding container ccf1242dc798e4da2cdfac4022669f3544dad2db0b3a024ab23669816258bf0b: Status 404 returned error can't find the container with id ccf1242dc798e4da2cdfac4022669f3544dad2db0b3a024ab23669816258bf0b
Jul 24 17:14:54.884238 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:54.884216    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:14:55.451434 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:55.451331    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9651e0f7-6bb6685777-sdxnw" event={"ID":"0f41db40-8e20-4034-8b3d-7ee308d5c277","Type":"ContainerStarted","Data":"a65a2457b6360e711b5292c3325fef092c51b61990bf3dee991fdb8bca0a0a5b"}
Jul 24 17:14:55.451434 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:55.451428    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9651e0f7-6bb6685777-sdxnw" event={"ID":"0f41db40-8e20-4034-8b3d-7ee308d5c277","Type":"ContainerStarted","Data":"a9b9a35af0f960f8e288aaff2227d80c09e4a4789756e8f17f26fec3cd0901a0"}
Jul 24 17:14:55.451673 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:55.451462    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-9651e0f7-6bb6685777-sdxnw"
Jul 24 17:14:55.453122 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:55.453097    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9651e0f7-65d5b9b5cb-t77ph" event={"ID":"cf87e19d-94b7-4394-bd82-3aace42bc0e8","Type":"ContainerStarted","Data":"e15f51568ab6329b222145a65d5de973f451d086b2aac7f3cca44e5c718697e0"}
Jul 24 17:14:55.453257 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:55.453135    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9651e0f7-65d5b9b5cb-t77ph" event={"ID":"cf87e19d-94b7-4394-bd82-3aace42bc0e8","Type":"ContainerStarted","Data":"ccf1242dc798e4da2cdfac4022669f3544dad2db0b3a024ab23669816258bf0b"}
Jul 24 17:14:55.453257 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:55.453215    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9651e0f7-65d5b9b5cb-t77ph"
Jul 24 17:14:55.454913 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:55.454893    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9651e0f7-6b7fb7768c-vrwgv" event={"ID":"e1284452-e388-432a-95d7-85bf333a0176","Type":"ContainerStarted","Data":"da62481261d23d17d390d29f168b275ff59cd0de5ceb625661501e04995f6335"}
Jul 24 17:14:55.455037 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:55.454947    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9651e0f7-6b7fb7768c-vrwgv" event={"ID":"e1284452-e388-432a-95d7-85bf333a0176","Type":"ContainerStarted","Data":"b0d89a44dc1d1fe5c9a82a1b4d69fc6a42083090d3f60e393a08b614e0968365"}
Jul 24 17:14:55.455037 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:55.455005    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9651e0f7-6b7fb7768c-vrwgv"
Jul 24 17:14:55.468594 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:55.468551    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-9651e0f7-6bb6685777-sdxnw" podStartSLOduration=1.468538962 podStartE2EDuration="1.468538962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:14:55.466227956 +0000 UTC m=+1918.108191862" watchObservedRunningTime="2026-07-24 17:14:55.468538962 +0000 UTC m=+1918.110502845"
Jul 24 17:14:55.484297 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:55.484249    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9651e0f7-6b7fb7768c-vrwgv" podStartSLOduration=1.9143960020000002 podStartE2EDuration="2.484232529s" totalImagesPullingTime="569.836527ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:14:55.482157874 +0000 UTC m=+1918.124121763" watchObservedRunningTime="2026-07-24 17:14:55.484232529 +0000 UTC m=+1918.126196438"
Jul 24 17:14:55.497978 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:14:55.497920    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9651e0f7-65d5b9b5cb-t77ph" podStartSLOduration=1.497905882 podStartE2EDuration="1.497905882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:14:55.496214999 +0000 UTC m=+1918.138178899" watchObservedRunningTime="2026-07-24 17:14:55.497905882 +0000 UTC m=+1918.139869766"
Jul 24 17:15:01.465923 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:01.465894    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9651e0f7-6b7fb7768c-vrwgv"
Jul 24 17:15:06.462744 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:06.462707    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-9651e0f7-6bb6685777-sdxnw"
Jul 24 17:15:06.463286 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:06.462839    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9651e0f7-65d5b9b5cb-t77ph"
Jul 24 17:15:29.048262 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.048226    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9651e0f7-6b7fb7768c-vrwgv"]
Jul 24 17:15:29.048826 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.048484    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9651e0f7-6b7fb7768c-vrwgv" podUID="e1284452-e388-432a-95d7-85bf333a0176" containerName="maas-api" containerID="cri-o://da62481261d23d17d390d29f168b275ff59cd0de5ceb625661501e04995f6335" gracePeriod=30
Jul 24 17:15:29.074550 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.074520    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:15:29.092531 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.092500    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9651e0f7-6bb6685777-sdxnw"]
Jul 24 17:15:29.092756 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.092728    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-9651e0f7-6bb6685777-sdxnw" podUID="0f41db40-8e20-4034-8b3d-7ee308d5c277" containerName="payload-processing" containerID="cri-o://a65a2457b6360e711b5292c3325fef092c51b61990bf3dee991fdb8bca0a0a5b" gracePeriod=30
Jul 24 17:15:29.101198 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.101166    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9651e0f7-65d5b9b5cb-t77ph"]
Jul 24 17:15:29.101428 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.101400    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9651e0f7-65d5b9b5cb-t77ph" podUID="cf87e19d-94b7-4394-bd82-3aace42bc0e8" containerName="payload-pre-processing" containerID="cri-o://e15f51568ab6329b222145a65d5de973f451d086b2aac7f3cca44e5c718697e0" gracePeriod=30
Jul 24 17:15:29.137004 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.136971    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:15:29.197678 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.197645    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:15:29.394889 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.394852    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9651e0f7-6bb6685777-sdxnw"
Jul 24 17:15:29.398155 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.398127    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9651e0f7-65d5b9b5cb-t77ph"
Jul 24 17:15:29.401326 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.401308    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9651e0f7-6b7fb7768c-vrwgv"
Jul 24 17:15:29.447328 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.447292    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrfph\" (UniqueName: \"kubernetes.io/projected/cf87e19d-94b7-4394-bd82-3aace42bc0e8-kube-api-access-rrfph\") pod \"cf87e19d-94b7-4394-bd82-3aace42bc0e8\" (UID: \"cf87e19d-94b7-4394-bd82-3aace42bc0e8\") "
Jul 24 17:15:29.447523 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.447386    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1284452-e388-432a-95d7-85bf333a0176-maas-api-tls\") pod \"e1284452-e388-432a-95d7-85bf333a0176\" (UID: \"e1284452-e388-432a-95d7-85bf333a0176\") "
Jul 24 17:15:29.447523 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.447419    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf87e19d-94b7-4394-bd82-3aace42bc0e8-plugins-config-volume\") pod \"cf87e19d-94b7-4394-bd82-3aace42bc0e8\" (UID: \"cf87e19d-94b7-4394-bd82-3aace42bc0e8\") "
Jul 24 17:15:29.447523 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.447452    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f41db40-8e20-4034-8b3d-7ee308d5c277-plugins-config-volume\") pod \"0f41db40-8e20-4034-8b3d-7ee308d5c277\" (UID: \"0f41db40-8e20-4034-8b3d-7ee308d5c277\") "
Jul 24 17:15:29.447523 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.447477    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1284452-e388-432a-95d7-85bf333a0176-openshift-service-ca\") pod \"e1284452-e388-432a-95d7-85bf333a0176\" (UID: \"e1284452-e388-432a-95d7-85bf333a0176\") "
Jul 24 17:15:29.447523 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.447513    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shqz8\" (UniqueName: \"kubernetes.io/projected/0f41db40-8e20-4034-8b3d-7ee308d5c277-kube-api-access-shqz8\") pod \"0f41db40-8e20-4034-8b3d-7ee308d5c277\" (UID: \"0f41db40-8e20-4034-8b3d-7ee308d5c277\") "
Jul 24 17:15:29.447775 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.447536    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9qjz\" (UniqueName: \"kubernetes.io/projected/e1284452-e388-432a-95d7-85bf333a0176-kube-api-access-v9qjz\") pod \"e1284452-e388-432a-95d7-85bf333a0176\" (UID: \"e1284452-e388-432a-95d7-85bf333a0176\") "
Jul 24 17:15:29.448188 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.448096    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f41db40-8e20-4034-8b3d-7ee308d5c277-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0f41db40-8e20-4034-8b3d-7ee308d5c277" (UID: "0f41db40-8e20-4034-8b3d-7ee308d5c277"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:15:29.448291 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.448248    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1284452-e388-432a-95d7-85bf333a0176-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e1284452-e388-432a-95d7-85bf333a0176" (UID: "e1284452-e388-432a-95d7-85bf333a0176"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:15:29.448560 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.448529    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf87e19d-94b7-4394-bd82-3aace42bc0e8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cf87e19d-94b7-4394-bd82-3aace42bc0e8" (UID: "cf87e19d-94b7-4394-bd82-3aace42bc0e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:15:29.449747 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.449720    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf87e19d-94b7-4394-bd82-3aace42bc0e8-kube-api-access-rrfph" (OuterVolumeSpecName: "kube-api-access-rrfph") pod "cf87e19d-94b7-4394-bd82-3aace42bc0e8" (UID: "cf87e19d-94b7-4394-bd82-3aace42bc0e8"). InnerVolumeSpecName "kube-api-access-rrfph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:15:29.450193 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.450173    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1284452-e388-432a-95d7-85bf333a0176-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e1284452-e388-432a-95d7-85bf333a0176" (UID: "e1284452-e388-432a-95d7-85bf333a0176"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:15:29.450436 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.450405    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f41db40-8e20-4034-8b3d-7ee308d5c277-kube-api-access-shqz8" (OuterVolumeSpecName: "kube-api-access-shqz8") pod "0f41db40-8e20-4034-8b3d-7ee308d5c277" (UID: "0f41db40-8e20-4034-8b3d-7ee308d5c277"). InnerVolumeSpecName "kube-api-access-shqz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:15:29.450532 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.450508    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1284452-e388-432a-95d7-85bf333a0176-kube-api-access-v9qjz" (OuterVolumeSpecName: "kube-api-access-v9qjz") pod "e1284452-e388-432a-95d7-85bf333a0176" (UID: "e1284452-e388-432a-95d7-85bf333a0176"). InnerVolumeSpecName "kube-api-access-v9qjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:15:29.548862 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.548811    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrfph\" (UniqueName: \"kubernetes.io/projected/cf87e19d-94b7-4394-bd82-3aace42bc0e8-kube-api-access-rrfph\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:15:29.548862 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.548856    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1284452-e388-432a-95d7-85bf333a0176-maas-api-tls\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:15:29.548862 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.548866    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf87e19d-94b7-4394-bd82-3aace42bc0e8-plugins-config-volume\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:15:29.548862 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.548875    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f41db40-8e20-4034-8b3d-7ee308d5c277-plugins-config-volume\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:15:29.548862 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.548885    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1284452-e388-432a-95d7-85bf333a0176-openshift-service-ca\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:15:29.549222 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.548893    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shqz8\" (UniqueName: \"kubernetes.io/projected/0f41db40-8e20-4034-8b3d-7ee308d5c277-kube-api-access-shqz8\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:15:29.549222 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.548901    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9qjz\" (UniqueName: \"kubernetes.io/projected/e1284452-e388-432a-95d7-85bf333a0176-kube-api-access-v9qjz\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:15:29.628326 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.628226    2563 generic.go:356] "Generic (PLEG): container finished" podID="cf87e19d-94b7-4394-bd82-3aace42bc0e8" containerID="e15f51568ab6329b222145a65d5de973f451d086b2aac7f3cca44e5c718697e0" exitCode=0
Jul 24 17:15:29.628326 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.628290    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9651e0f7-65d5b9b5cb-t77ph"
Jul 24 17:15:29.628571 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.628313    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9651e0f7-65d5b9b5cb-t77ph" event={"ID":"cf87e19d-94b7-4394-bd82-3aace42bc0e8","Type":"ContainerDied","Data":"e15f51568ab6329b222145a65d5de973f451d086b2aac7f3cca44e5c718697e0"}
Jul 24 17:15:29.628571 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.628401    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9651e0f7-65d5b9b5cb-t77ph" event={"ID":"cf87e19d-94b7-4394-bd82-3aace42bc0e8","Type":"ContainerDied","Data":"ccf1242dc798e4da2cdfac4022669f3544dad2db0b3a024ab23669816258bf0b"}
Jul 24 17:15:29.628571 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.628431    2563 scope.go:117] "RemoveContainer" containerID="e15f51568ab6329b222145a65d5de973f451d086b2aac7f3cca44e5c718697e0"
Jul 24 17:15:29.629759 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.629736    2563 generic.go:356] "Generic (PLEG): container finished" podID="e1284452-e388-432a-95d7-85bf333a0176" containerID="da62481261d23d17d390d29f168b275ff59cd0de5ceb625661501e04995f6335" exitCode=0
Jul 24 17:15:29.629831 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.629806    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9651e0f7-6b7fb7768c-vrwgv"
Jul 24 17:15:29.629885 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.629846    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9651e0f7-6b7fb7768c-vrwgv" event={"ID":"e1284452-e388-432a-95d7-85bf333a0176","Type":"ContainerDied","Data":"da62481261d23d17d390d29f168b275ff59cd0de5ceb625661501e04995f6335"}
Jul 24 17:15:29.629956 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.629895    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9651e0f7-6b7fb7768c-vrwgv" event={"ID":"e1284452-e388-432a-95d7-85bf333a0176","Type":"ContainerDied","Data":"b0d89a44dc1d1fe5c9a82a1b4d69fc6a42083090d3f60e393a08b614e0968365"}
Jul 24 17:15:29.631406 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.631383    2563 generic.go:356] "Generic (PLEG): container finished" podID="0f41db40-8e20-4034-8b3d-7ee308d5c277" containerID="a65a2457b6360e711b5292c3325fef092c51b61990bf3dee991fdb8bca0a0a5b" exitCode=0
Jul 24 17:15:29.631530 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.631418    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9651e0f7-6bb6685777-sdxnw" event={"ID":"0f41db40-8e20-4034-8b3d-7ee308d5c277","Type":"ContainerDied","Data":"a65a2457b6360e711b5292c3325fef092c51b61990bf3dee991fdb8bca0a0a5b"}
Jul 24 17:15:29.631530 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.631434    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9651e0f7-6bb6685777-sdxnw"
Jul 24 17:15:29.631530 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.631457    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9651e0f7-6bb6685777-sdxnw" event={"ID":"0f41db40-8e20-4034-8b3d-7ee308d5c277","Type":"ContainerDied","Data":"a9b9a35af0f960f8e288aaff2227d80c09e4a4789756e8f17f26fec3cd0901a0"}
Jul 24 17:15:29.639426 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.639394    2563 scope.go:117] "RemoveContainer" containerID="e15f51568ab6329b222145a65d5de973f451d086b2aac7f3cca44e5c718697e0"
Jul 24 17:15:29.639722 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:15:29.639694    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e15f51568ab6329b222145a65d5de973f451d086b2aac7f3cca44e5c718697e0\": container with ID starting with e15f51568ab6329b222145a65d5de973f451d086b2aac7f3cca44e5c718697e0 not found: ID does not exist" containerID="e15f51568ab6329b222145a65d5de973f451d086b2aac7f3cca44e5c718697e0"
Jul 24 17:15:29.639812 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.639731    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e15f51568ab6329b222145a65d5de973f451d086b2aac7f3cca44e5c718697e0"} err="failed to get container status \"e15f51568ab6329b222145a65d5de973f451d086b2aac7f3cca44e5c718697e0\": rpc error: code = NotFound desc = could not find container \"e15f51568ab6329b222145a65d5de973f451d086b2aac7f3cca44e5c718697e0\": container with ID starting with e15f51568ab6329b222145a65d5de973f451d086b2aac7f3cca44e5c718697e0 not found: ID does not exist"
Jul 24 17:15:29.639812 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.639754    2563 scope.go:117] "RemoveContainer" containerID="da62481261d23d17d390d29f168b275ff59cd0de5ceb625661501e04995f6335"
Jul 24 17:15:29.648147 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.648130    2563 scope.go:117] "RemoveContainer" containerID="da62481261d23d17d390d29f168b275ff59cd0de5ceb625661501e04995f6335"
Jul 24 17:15:29.648417 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:15:29.648398    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da62481261d23d17d390d29f168b275ff59cd0de5ceb625661501e04995f6335\": container with ID starting with da62481261d23d17d390d29f168b275ff59cd0de5ceb625661501e04995f6335 not found: ID does not exist" containerID="da62481261d23d17d390d29f168b275ff59cd0de5ceb625661501e04995f6335"
Jul 24 17:15:29.648464 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.648424    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da62481261d23d17d390d29f168b275ff59cd0de5ceb625661501e04995f6335"} err="failed to get container status \"da62481261d23d17d390d29f168b275ff59cd0de5ceb625661501e04995f6335\": rpc error: code = NotFound desc = could not find container \"da62481261d23d17d390d29f168b275ff59cd0de5ceb625661501e04995f6335\": container with ID starting with da62481261d23d17d390d29f168b275ff59cd0de5ceb625661501e04995f6335 not found: ID does not exist"
Jul 24 17:15:29.648464 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.648441    2563 scope.go:117] "RemoveContainer" containerID="a65a2457b6360e711b5292c3325fef092c51b61990bf3dee991fdb8bca0a0a5b"
Jul 24 17:15:29.654380 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.654346    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9651e0f7-6b7fb7768c-vrwgv"]
Jul 24 17:15:29.657333 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.657314    2563 scope.go:117] "RemoveContainer" containerID="a65a2457b6360e711b5292c3325fef092c51b61990bf3dee991fdb8bca0a0a5b"
Jul 24 17:15:29.657616 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:15:29.657588    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a65a2457b6360e711b5292c3325fef092c51b61990bf3dee991fdb8bca0a0a5b\": container with ID starting with a65a2457b6360e711b5292c3325fef092c51b61990bf3dee991fdb8bca0a0a5b not found: ID does not exist" containerID="a65a2457b6360e711b5292c3325fef092c51b61990bf3dee991fdb8bca0a0a5b"
Jul 24 17:15:29.657713 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.657625    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a65a2457b6360e711b5292c3325fef092c51b61990bf3dee991fdb8bca0a0a5b"} err="failed to get container status \"a65a2457b6360e711b5292c3325fef092c51b61990bf3dee991fdb8bca0a0a5b\": rpc error: code = NotFound desc = could not find container \"a65a2457b6360e711b5292c3325fef092c51b61990bf3dee991fdb8bca0a0a5b\": container with ID starting with a65a2457b6360e711b5292c3325fef092c51b61990bf3dee991fdb8bca0a0a5b not found: ID does not exist"
Jul 24 17:15:29.658317 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.658300    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9651e0f7-6b7fb7768c-vrwgv"]
Jul 24 17:15:29.667652 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.667630    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9651e0f7-65d5b9b5cb-t77ph"]
Jul 24 17:15:29.671581 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.671559    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9651e0f7-65d5b9b5cb-t77ph"]
Jul 24 17:15:29.681819 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.681739    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9651e0f7-6bb6685777-sdxnw"]
Jul 24 17:15:29.683855 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:29.683833    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9651e0f7-6bb6685777-sdxnw"]
Jul 24 17:15:30.003629 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:30.003598    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f41db40-8e20-4034-8b3d-7ee308d5c277" path="/var/lib/kubelet/pods/0f41db40-8e20-4034-8b3d-7ee308d5c277/volumes"
Jul 24 17:15:30.003993 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:30.003979    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf87e19d-94b7-4394-bd82-3aace42bc0e8" path="/var/lib/kubelet/pods/cf87e19d-94b7-4394-bd82-3aace42bc0e8/volumes"
Jul 24 17:15:30.004333 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:30.004321    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1284452-e388-432a-95d7-85bf333a0176" path="/var/lib/kubelet/pods/e1284452-e388-432a-95d7-85bf333a0176/volumes"
Jul 24 17:15:33.565490 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:33.565461    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:15:33.721798 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:33.721763    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:15:33.952893 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:33.952857    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:15:34.173673 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:34.173636    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:15:34.499887 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:34.499854    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:15:34.905007 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:34.904974    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:15:36.654154 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:36.654090    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:15:36.812269 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:36.812233    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9300b6a3-75f7dfdbd9-7kxcz"]
Jul 24 17:15:36.812628 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:36.812615    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f41db40-8e20-4034-8b3d-7ee308d5c277" containerName="payload-processing"
Jul 24 17:15:36.812672 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:36.812629    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f41db40-8e20-4034-8b3d-7ee308d5c277" containerName="payload-processing"
Jul 24 17:15:36.812672 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:36.812637    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1284452-e388-432a-95d7-85bf333a0176" containerName="maas-api"
Jul 24 17:15:36.812672 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:36.812643    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1284452-e388-432a-95d7-85bf333a0176" containerName="maas-api"
Jul 24 17:15:36.812672 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:36.812658    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf87e19d-94b7-4394-bd82-3aace42bc0e8" containerName="payload-pre-processing"
Jul 24 17:15:36.812672 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:36.812664    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf87e19d-94b7-4394-bd82-3aace42bc0e8" containerName="payload-pre-processing"
Jul 24 17:15:36.812828 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:36.812721    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1284452-e388-432a-95d7-85bf333a0176" containerName="maas-api"
Jul 24 17:15:36.812828 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:36.812729    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf87e19d-94b7-4394-bd82-3aace42bc0e8" containerName="payload-pre-processing"
Jul 24 17:15:36.812828 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:36.812737    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f41db40-8e20-4034-8b3d-7ee308d5c277" containerName="payload-processing"
Jul 24 17:15:36.815841 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:36.815825    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9300b6a3-75f7dfdbd9-7kxcz"
Jul 24 17:15:36.816775 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:36.816754    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:15:36.819518 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:36.819499    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-9300b6a3-serving-cert\""
Jul 24 17:15:36.843950 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:36.843902    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9300b6a3-75f7dfdbd9-7kxcz"]
Jul 24 17:15:36.910078 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:36.909993    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:15:36.915224 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:36.915194    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bae67e99-d96b-40d9-9a2e-53f843a13c6b-openshift-service-ca\") pod \"maas-api-e2e-ait-9300b6a3-75f7dfdbd9-7kxcz\" (UID: \"bae67e99-d96b-40d9-9a2e-53f843a13c6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9300b6a3-75f7dfdbd9-7kxcz"
Jul 24 17:15:36.915361 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:36.915263    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bae67e99-d96b-40d9-9a2e-53f843a13c6b-maas-api-tls\") pod \"maas-api-e2e-ait-9300b6a3-75f7dfdbd9-7kxcz\" (UID: \"bae67e99-d96b-40d9-9a2e-53f843a13c6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9300b6a3-75f7dfdbd9-7kxcz"
Jul 24 17:15:36.915361 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:36.915299    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72njq\" (UniqueName: \"kubernetes.io/projected/bae67e99-d96b-40d9-9a2e-53f843a13c6b-kube-api-access-72njq\") pod \"maas-api-e2e-ait-9300b6a3-75f7dfdbd9-7kxcz\" (UID: \"bae67e99-d96b-40d9-9a2e-53f843a13c6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9300b6a3-75f7dfdbd9-7kxcz"
Jul 24 17:15:37.016765 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:37.016723    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72njq\" (UniqueName: \"kubernetes.io/projected/bae67e99-d96b-40d9-9a2e-53f843a13c6b-kube-api-access-72njq\") pod \"maas-api-e2e-ait-9300b6a3-75f7dfdbd9-7kxcz\" (UID: \"bae67e99-d96b-40d9-9a2e-53f843a13c6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9300b6a3-75f7dfdbd9-7kxcz"
Jul 24 17:15:37.016976 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:37.016891    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bae67e99-d96b-40d9-9a2e-53f843a13c6b-openshift-service-ca\") pod \"maas-api-e2e-ait-9300b6a3-75f7dfdbd9-7kxcz\" (UID: \"bae67e99-d96b-40d9-9a2e-53f843a13c6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9300b6a3-75f7dfdbd9-7kxcz"
Jul 24 17:15:37.017033 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:37.016998    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bae67e99-d96b-40d9-9a2e-53f843a13c6b-maas-api-tls\") pod \"maas-api-e2e-ait-9300b6a3-75f7dfdbd9-7kxcz\" (UID: \"bae67e99-d96b-40d9-9a2e-53f843a13c6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9300b6a3-75f7dfdbd9-7kxcz"
Jul 24 17:15:37.017118 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:15:37.017105    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-9300b6a3-serving-cert: secret "maas-api-e2e-ait-9300b6a3-serving-cert" not found
Jul 24 17:15:37.017168 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:15:37.017159    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bae67e99-d96b-40d9-9a2e-53f843a13c6b-maas-api-tls podName:bae67e99-d96b-40d9-9a2e-53f843a13c6b nodeName:}" failed. No retries permitted until 2026-07-24 17:15:37.517143721 +0000 UTC m=+1960.159107583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bae67e99-d96b-40d9-9a2e-53f843a13c6b-maas-api-tls") pod "maas-api-e2e-ait-9300b6a3-75f7dfdbd9-7kxcz" (UID: "bae67e99-d96b-40d9-9a2e-53f843a13c6b") : secret "maas-api-e2e-ait-9300b6a3-serving-cert" not found
Jul 24 17:15:37.017554 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:37.017535    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bae67e99-d96b-40d9-9a2e-53f843a13c6b-openshift-service-ca\") pod \"maas-api-e2e-ait-9300b6a3-75f7dfdbd9-7kxcz\" (UID: \"bae67e99-d96b-40d9-9a2e-53f843a13c6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9300b6a3-75f7dfdbd9-7kxcz"
Jul 24 17:15:37.029826 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:37.029797    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72njq\" (UniqueName: \"kubernetes.io/projected/bae67e99-d96b-40d9-9a2e-53f843a13c6b-kube-api-access-72njq\") pod \"maas-api-e2e-ait-9300b6a3-75f7dfdbd9-7kxcz\" (UID: \"bae67e99-d96b-40d9-9a2e-53f843a13c6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9300b6a3-75f7dfdbd9-7kxcz"
Jul 24 17:15:37.375097 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:37.374990    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:15:37.472960 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:37.472902    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:15:37.476689 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:37.476660    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9300b6a3-646cd48f68-cb5k9"]
Jul 24 17:15:37.480245 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:37.480222    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9300b6a3-646cd48f68-cb5k9"
Jul 24 17:15:37.482802 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:37.482781    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9300b6a3\""
Jul 24 17:15:37.493303 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:37.493281    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9300b6a3-646cd48f68-cb5k9"]
Jul 24 17:15:37.504597 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:37.504568    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9300b6a3-c77d44cb-5n7hh"]
Jul 24 17:15:37.508337 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:37.508310    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9300b6a3-c77d44cb-5n7hh"
Jul 24 17:15:37.510876 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:37.510857    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9300b6a3-dockercfg-tb6ct\""
Jul 24 17:15:37.517335 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:37.517309    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9300b6a3-c77d44cb-5n7hh"]
Jul 24 17:15:37.522165 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:37.522138    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bae67e99-d96b-40d9-9a2e-53f843a13c6b-maas-api-tls\") pod \"maas-api-e2e-ait-9300b6a3-75f7dfdbd9-7kxcz\" (UID: \"bae67e99-d96b-40d9-9a2e-53f843a13c6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9300b6a3-75f7dfdbd9-7kxcz"
Jul 24 17:15:37.524501 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:37.524475    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bae67e99-d96b-40d9-9a2e-53f843a13c6b-maas-api-tls\") pod \"maas-api-e2e-ait-9300b6a3-75f7dfdbd9-7kxcz\" (UID: \"bae67e99-d96b-40d9-9a2e-53f843a13c6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9300b6a3-75f7dfdbd9-7kxcz"
Jul 24 17:15:37.623572 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:37.623053    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1bd7192-0408-4e97-bf92-6a2a87106c55-plugins-config-volume\") pod \"payload-processing-e2e-ait-9300b6a3-c77d44cb-5n7hh\" (UID: \"e1bd7192-0408-4e97-bf92-6a2a87106c55\") " pod="openshift-ingress/payload-processing-e2e-ait-9300b6a3-c77d44cb-5n7hh"
Jul 24 17:15:37.623572 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:37.623211    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvvzq\" (UniqueName: \"kubernetes.io/projected/e1bd7192-0408-4e97-bf92-6a2a87106c55-kube-api-access-jvvzq\") pod \"payload-processing-e2e-ait-9300b6a3-c77d44cb-5n7hh\" (UID: \"e1bd7192-0408-4e97-bf92-6a2a87106c55\") " pod="openshift-ingress/payload-processing-e2e-ait-9300b6a3-c77d44cb-5n7hh"
Jul 24 17:15:37.623572 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:37.623263    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vpx2\" (UniqueName: \"kubernetes.io/projected/c8a023f3-a40a-4bbe-8eb8-fbfbf1efc8de-kube-api-access-9vpx2\") pod \"payload-pre-processing-e2e-ait-9300b6a3-646cd48f68-cb5k9\" (UID: \"c8a023f3-a40a-4bbe-8eb8-fbfbf1efc8de\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9300b6a3-646cd48f68-cb5k9"
Jul 24 17:15:37.623572 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:37.623297    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8a023f3-a40a-4bbe-8eb8-fbfbf1efc8de-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9300b6a3-646cd48f68-cb5k9\" (UID: \"c8a023f3-a40a-4bbe-8eb8-fbfbf1efc8de\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9300b6a3-646cd48f68-cb5k9"
Jul 24 17:15:37.724799 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:37.724763    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvvzq\" (UniqueName: \"kubernetes.io/projected/e1bd7192-0408-4e97-bf92-6a2a87106c55-kube-api-access-jvvzq\") pod \"payload-processing-e2e-ait-9300b6a3-c77d44cb-5n7hh\" (UID: \"e1bd7192-0408-4e97-bf92-6a2a87106c55\") " pod="openshift-ingress/payload-processing-e2e-ait-9300b6a3-c77d44cb-5n7hh"
Jul 24 17:15:37.724799 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:37.724809    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vpx2\" (UniqueName: \"kubernetes.io/projected/c8a023f3-a40a-4bbe-8eb8-fbfbf1efc8de-kube-api-access-9vpx2\") pod \"payload-pre-processing-e2e-ait-9300b6a3-646cd48f68-cb5k9\" (UID: \"c8a023f3-a40a-4bbe-8eb8-fbfbf1efc8de\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9300b6a3-646cd48f68-cb5k9"
Jul 24 17:15:37.725299 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:37.724836    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8a023f3-a40a-4bbe-8eb8-fbfbf1efc8de-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9300b6a3-646cd48f68-cb5k9\" (UID: \"c8a023f3-a40a-4bbe-8eb8-fbfbf1efc8de\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9300b6a3-646cd48f68-cb5k9"
Jul 24 17:15:37.725299 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:37.724890    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1bd7192-0408-4e97-bf92-6a2a87106c55-plugins-config-volume\") pod \"payload-processing-e2e-ait-9300b6a3-c77d44cb-5n7hh\" (UID: \"e1bd7192-0408-4e97-bf92-6a2a87106c55\") " pod="openshift-ingress/payload-processing-e2e-ait-9300b6a3-c77d44cb-5n7hh"
Jul 24 17:15:37.725630 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:37.725609    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1bd7192-0408-4e97-bf92-6a2a87106c55-plugins-config-volume\") pod \"payload-processing-e2e-ait-9300b6a3-c77d44cb-5n7hh\" (UID: \"e1bd7192-0408-4e97-bf92-6a2a87106c55\") " pod="openshift-ingress/payload-processing-e2e-ait-9300b6a3-c77d44cb-5n7hh"
Jul 24 17:15:37.725674 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:37.725635    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8a023f3-a40a-4bbe-8eb8-fbfbf1efc8de-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9300b6a3-646cd48f68-cb5k9\" (UID: \"c8a023f3-a40a-4bbe-8eb8-fbfbf1efc8de\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9300b6a3-646cd48f68-cb5k9"
Jul 24 17:15:37.725743 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:37.725731    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9300b6a3-75f7dfdbd9-7kxcz"
Jul 24 17:15:37.734303 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:37.734280    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vpx2\" (UniqueName: \"kubernetes.io/projected/c8a023f3-a40a-4bbe-8eb8-fbfbf1efc8de-kube-api-access-9vpx2\") pod \"payload-pre-processing-e2e-ait-9300b6a3-646cd48f68-cb5k9\" (UID: \"c8a023f3-a40a-4bbe-8eb8-fbfbf1efc8de\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9300b6a3-646cd48f68-cb5k9"
Jul 24 17:15:37.734429 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:37.734359    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvvzq\" (UniqueName: \"kubernetes.io/projected/e1bd7192-0408-4e97-bf92-6a2a87106c55-kube-api-access-jvvzq\") pod \"payload-processing-e2e-ait-9300b6a3-c77d44cb-5n7hh\" (UID: \"e1bd7192-0408-4e97-bf92-6a2a87106c55\") " pod="openshift-ingress/payload-processing-e2e-ait-9300b6a3-c77d44cb-5n7hh"
Jul 24 17:15:37.791507 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:37.791465    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9300b6a3-646cd48f68-cb5k9"
Jul 24 17:15:37.819588 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:37.819501    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9300b6a3-c77d44cb-5n7hh"
Jul 24 17:15:37.884157 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:37.884123    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9300b6a3-75f7dfdbd9-7kxcz"]
Jul 24 17:15:37.887719 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:15:37.887687    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae67e99_d96b_40d9_9a2e_53f843a13c6b.slice/crio-876f9dfbc765a50ee3fe5525b20e38d27aad25c3249b59f4aeaa03eaaf60e3c3 WatchSource:0}: Error finding container 876f9dfbc765a50ee3fe5525b20e38d27aad25c3249b59f4aeaa03eaaf60e3c3: Status 404 returned error can't find the container with id 876f9dfbc765a50ee3fe5525b20e38d27aad25c3249b59f4aeaa03eaaf60e3c3
Jul 24 17:15:37.977555 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:37.977518    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9300b6a3-646cd48f68-cb5k9"]
Jul 24 17:15:37.979901 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:15:37.979875    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a023f3_a40a_4bbe_8eb8_fbfbf1efc8de.slice/crio-613e72503718516e5ccae514faf73ca99a979aadcba104338f1dc452fa978103 WatchSource:0}: Error finding container 613e72503718516e5ccae514faf73ca99a979aadcba104338f1dc452fa978103: Status 404 returned error can't find the container with id 613e72503718516e5ccae514faf73ca99a979aadcba104338f1dc452fa978103
Jul 24 17:15:37.995878 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:37.995772    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9300b6a3-c77d44cb-5n7hh"]
Jul 24 17:15:37.998888 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:15:37.998786    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1bd7192_0408_4e97_bf92_6a2a87106c55.slice/crio-9f5b5bda4dfa09ef4ad14c18920552f6076a40091aab28f993b17549ce7a92f9 WatchSource:0}: Error finding container 9f5b5bda4dfa09ef4ad14c18920552f6076a40091aab28f993b17549ce7a92f9: Status 404 returned error can't find the container with id 9f5b5bda4dfa09ef4ad14c18920552f6076a40091aab28f993b17549ce7a92f9
Jul 24 17:15:38.679089 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:38.679049    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9300b6a3-75f7dfdbd9-7kxcz" event={"ID":"bae67e99-d96b-40d9-9a2e-53f843a13c6b","Type":"ContainerStarted","Data":"470bf7e93906dcd14319763699e22f92899e9087901d48838b7bb82c71be2efa"}
Jul 24 17:15:38.679304 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:38.679118    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9300b6a3-75f7dfdbd9-7kxcz" event={"ID":"bae67e99-d96b-40d9-9a2e-53f843a13c6b","Type":"ContainerStarted","Data":"876f9dfbc765a50ee3fe5525b20e38d27aad25c3249b59f4aeaa03eaaf60e3c3"}
Jul 24 17:15:38.679304 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:38.679150    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9300b6a3-75f7dfdbd9-7kxcz"
Jul 24 17:15:38.680575 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:38.680546    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9300b6a3-c77d44cb-5n7hh" event={"ID":"e1bd7192-0408-4e97-bf92-6a2a87106c55","Type":"ContainerStarted","Data":"2a4b4c1d9fb062981a3350f85b996413343b6622e85f45ebe26d38fd4b2ae361"}
Jul 24 17:15:38.680696 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:38.680601    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-9300b6a3-c77d44cb-5n7hh"
Jul 24 17:15:38.680696 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:38.680619    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9300b6a3-c77d44cb-5n7hh" event={"ID":"e1bd7192-0408-4e97-bf92-6a2a87106c55","Type":"ContainerStarted","Data":"9f5b5bda4dfa09ef4ad14c18920552f6076a40091aab28f993b17549ce7a92f9"}
Jul 24 17:15:38.682026 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:38.682006    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9300b6a3-646cd48f68-cb5k9" event={"ID":"c8a023f3-a40a-4bbe-8eb8-fbfbf1efc8de","Type":"ContainerStarted","Data":"998c31bb99ea0873dc31592274a3842cd9723c374233f29cc01823a1ee93aa5d"}
Jul 24 17:15:38.682117 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:38.682039    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9300b6a3-646cd48f68-cb5k9" event={"ID":"c8a023f3-a40a-4bbe-8eb8-fbfbf1efc8de","Type":"ContainerStarted","Data":"613e72503718516e5ccae514faf73ca99a979aadcba104338f1dc452fa978103"}
Jul 24 17:15:38.682169 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:38.682152    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9300b6a3-646cd48f68-cb5k9"
Jul 24 17:15:38.698992 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:38.698947    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9300b6a3-75f7dfdbd9-7kxcz" podStartSLOduration=2.358501153 podStartE2EDuration="2.698917578s" totalImagesPullingTime="340.416425ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:15:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:15:38.696811034 +0000 UTC m=+1961.338774912" watchObservedRunningTime="2026-07-24 17:15:38.698917578 +0000 UTC m=+1961.340881461"
Jul 24 17:15:38.713961 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:38.713899    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-9300b6a3-c77d44cb-5n7hh" podStartSLOduration=1.7138824719999999 podStartE2EDuration="1.713882472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:15:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:15:38.712993806 +0000 UTC m=+1961.354957711" watchObservedRunningTime="2026-07-24 17:15:38.713882472 +0000 UTC m=+1961.355846410"
Jul 24 17:15:38.729182 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:38.729137    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-9300b6a3-646cd48f68-cb5k9" podStartSLOduration=1.729121659 podStartE2EDuration="1.729121659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:15:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:15:38.72873669 +0000 UTC m=+1961.370700590" watchObservedRunningTime="2026-07-24 17:15:38.729121659 +0000 UTC m=+1961.371085544"
Jul 24 17:15:44.692717 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:44.692687    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9300b6a3-75f7dfdbd9-7kxcz"
Jul 24 17:15:49.689128 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:49.689082    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-9300b6a3-c77d44cb-5n7hh"
Jul 24 17:15:49.689545 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:15:49.689275    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9300b6a3-646cd48f68-cb5k9"
Jul 24 17:16:12.789565 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:12.789529    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9300b6a3-75f7dfdbd9-7kxcz"]
Jul 24 17:16:12.790464 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:12.790428    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9300b6a3-75f7dfdbd9-7kxcz" podUID="bae67e99-d96b-40d9-9a2e-53f843a13c6b" containerName="maas-api" containerID="cri-o://470bf7e93906dcd14319763699e22f92899e9087901d48838b7bb82c71be2efa" gracePeriod=30
Jul 24 17:16:12.824974 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:12.824948    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:16:12.845551 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:12.845521    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9300b6a3-c77d44cb-5n7hh"]
Jul 24 17:16:12.845794 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:12.845766    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-9300b6a3-c77d44cb-5n7hh" podUID="e1bd7192-0408-4e97-bf92-6a2a87106c55" containerName="payload-processing" containerID="cri-o://2a4b4c1d9fb062981a3350f85b996413343b6622e85f45ebe26d38fd4b2ae361" gracePeriod=30
Jul 24 17:16:12.853717 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:12.853686    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9300b6a3-646cd48f68-cb5k9"]
Jul 24 17:16:12.853966 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:12.853921    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-9300b6a3-646cd48f68-cb5k9" podUID="c8a023f3-a40a-4bbe-8eb8-fbfbf1efc8de" containerName="payload-pre-processing" containerID="cri-o://998c31bb99ea0873dc31592274a3842cd9723c374233f29cc01823a1ee93aa5d" gracePeriod=30
Jul 24 17:16:12.890020 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:12.889920    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:16:12.946852 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:12.946825    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:16:13.138347 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.138318    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9300b6a3-c77d44cb-5n7hh"
Jul 24 17:16:13.141160 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.141139    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9300b6a3-75f7dfdbd9-7kxcz"
Jul 24 17:16:13.159457 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.159430    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9300b6a3-646cd48f68-cb5k9"
Jul 24 17:16:13.245652 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.245619    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vpx2\" (UniqueName: \"kubernetes.io/projected/c8a023f3-a40a-4bbe-8eb8-fbfbf1efc8de-kube-api-access-9vpx2\") pod \"c8a023f3-a40a-4bbe-8eb8-fbfbf1efc8de\" (UID: \"c8a023f3-a40a-4bbe-8eb8-fbfbf1efc8de\") "
Jul 24 17:16:13.245807 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.245685    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bae67e99-d96b-40d9-9a2e-53f843a13c6b-maas-api-tls\") pod \"bae67e99-d96b-40d9-9a2e-53f843a13c6b\" (UID: \"bae67e99-d96b-40d9-9a2e-53f843a13c6b\") "
Jul 24 17:16:13.245807 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.245713    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bae67e99-d96b-40d9-9a2e-53f843a13c6b-openshift-service-ca\") pod \"bae67e99-d96b-40d9-9a2e-53f843a13c6b\" (UID: \"bae67e99-d96b-40d9-9a2e-53f843a13c6b\") "
Jul 24 17:16:13.245807 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.245738    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8a023f3-a40a-4bbe-8eb8-fbfbf1efc8de-plugins-config-volume\") pod \"c8a023f3-a40a-4bbe-8eb8-fbfbf1efc8de\" (UID: \"c8a023f3-a40a-4bbe-8eb8-fbfbf1efc8de\") "
Jul 24 17:16:13.245807 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.245754    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvvzq\" (UniqueName: \"kubernetes.io/projected/e1bd7192-0408-4e97-bf92-6a2a87106c55-kube-api-access-jvvzq\") pod \"e1bd7192-0408-4e97-bf92-6a2a87106c55\" (UID: \"e1bd7192-0408-4e97-bf92-6a2a87106c55\") "
Jul 24 17:16:13.245807 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.245779    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1bd7192-0408-4e97-bf92-6a2a87106c55-plugins-config-volume\") pod \"e1bd7192-0408-4e97-bf92-6a2a87106c55\" (UID: \"e1bd7192-0408-4e97-bf92-6a2a87106c55\") "
Jul 24 17:16:13.246184 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.245827    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72njq\" (UniqueName: \"kubernetes.io/projected/bae67e99-d96b-40d9-9a2e-53f843a13c6b-kube-api-access-72njq\") pod \"bae67e99-d96b-40d9-9a2e-53f843a13c6b\" (UID: \"bae67e99-d96b-40d9-9a2e-53f843a13c6b\") "
Jul 24 17:16:13.246244 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.246185    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bae67e99-d96b-40d9-9a2e-53f843a13c6b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bae67e99-d96b-40d9-9a2e-53f843a13c6b" (UID: "bae67e99-d96b-40d9-9a2e-53f843a13c6b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:16:13.246244 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.246201    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8a023f3-a40a-4bbe-8eb8-fbfbf1efc8de-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c8a023f3-a40a-4bbe-8eb8-fbfbf1efc8de" (UID: "c8a023f3-a40a-4bbe-8eb8-fbfbf1efc8de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:16:13.246244 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.246210    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1bd7192-0408-4e97-bf92-6a2a87106c55-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e1bd7192-0408-4e97-bf92-6a2a87106c55" (UID: "e1bd7192-0408-4e97-bf92-6a2a87106c55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:16:13.247868 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.247841    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bae67e99-d96b-40d9-9a2e-53f843a13c6b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bae67e99-d96b-40d9-9a2e-53f843a13c6b" (UID: "bae67e99-d96b-40d9-9a2e-53f843a13c6b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:16:13.248100 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.247961    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bae67e99-d96b-40d9-9a2e-53f843a13c6b-kube-api-access-72njq" (OuterVolumeSpecName: "kube-api-access-72njq") pod "bae67e99-d96b-40d9-9a2e-53f843a13c6b" (UID: "bae67e99-d96b-40d9-9a2e-53f843a13c6b"). InnerVolumeSpecName "kube-api-access-72njq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:16:13.248265 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.248247    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8a023f3-a40a-4bbe-8eb8-fbfbf1efc8de-kube-api-access-9vpx2" (OuterVolumeSpecName: "kube-api-access-9vpx2") pod "c8a023f3-a40a-4bbe-8eb8-fbfbf1efc8de" (UID: "c8a023f3-a40a-4bbe-8eb8-fbfbf1efc8de"). InnerVolumeSpecName "kube-api-access-9vpx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:16:13.248311 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.248267    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1bd7192-0408-4e97-bf92-6a2a87106c55-kube-api-access-jvvzq" (OuterVolumeSpecName: "kube-api-access-jvvzq") pod "e1bd7192-0408-4e97-bf92-6a2a87106c55" (UID: "e1bd7192-0408-4e97-bf92-6a2a87106c55"). InnerVolumeSpecName "kube-api-access-jvvzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:16:13.347499 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.347414    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8a023f3-a40a-4bbe-8eb8-fbfbf1efc8de-plugins-config-volume\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:16:13.347499 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.347444    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvvzq\" (UniqueName: \"kubernetes.io/projected/e1bd7192-0408-4e97-bf92-6a2a87106c55-kube-api-access-jvvzq\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:16:13.347499 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.347458    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1bd7192-0408-4e97-bf92-6a2a87106c55-plugins-config-volume\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:16:13.347499 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.347470    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72njq\" (UniqueName: \"kubernetes.io/projected/bae67e99-d96b-40d9-9a2e-53f843a13c6b-kube-api-access-72njq\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:16:13.347499 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.347481    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vpx2\" (UniqueName: \"kubernetes.io/projected/c8a023f3-a40a-4bbe-8eb8-fbfbf1efc8de-kube-api-access-9vpx2\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:16:13.347499 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.347495    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bae67e99-d96b-40d9-9a2e-53f843a13c6b-maas-api-tls\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:16:13.347795 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.347510    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bae67e99-d96b-40d9-9a2e-53f843a13c6b-openshift-service-ca\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:16:13.861776 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.861736    2563 generic.go:356] "Generic (PLEG): container finished" podID="e1bd7192-0408-4e97-bf92-6a2a87106c55" containerID="2a4b4c1d9fb062981a3350f85b996413343b6622e85f45ebe26d38fd4b2ae361" exitCode=0
Jul 24 17:16:13.862244 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.861784    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9300b6a3-c77d44cb-5n7hh"
Jul 24 17:16:13.862244 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.861821    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9300b6a3-c77d44cb-5n7hh" event={"ID":"e1bd7192-0408-4e97-bf92-6a2a87106c55","Type":"ContainerDied","Data":"2a4b4c1d9fb062981a3350f85b996413343b6622e85f45ebe26d38fd4b2ae361"}
Jul 24 17:16:13.862244 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.861908    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9300b6a3-c77d44cb-5n7hh" event={"ID":"e1bd7192-0408-4e97-bf92-6a2a87106c55","Type":"ContainerDied","Data":"9f5b5bda4dfa09ef4ad14c18920552f6076a40091aab28f993b17549ce7a92f9"}
Jul 24 17:16:13.862244 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.861965    2563 scope.go:117] "RemoveContainer" containerID="2a4b4c1d9fb062981a3350f85b996413343b6622e85f45ebe26d38fd4b2ae361"
Jul 24 17:16:13.866568 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.866545    2563 generic.go:356] "Generic (PLEG): container finished" podID="c8a023f3-a40a-4bbe-8eb8-fbfbf1efc8de" containerID="998c31bb99ea0873dc31592274a3842cd9723c374233f29cc01823a1ee93aa5d" exitCode=0
Jul 24 17:16:13.866680 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.866572    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9300b6a3-646cd48f68-cb5k9" event={"ID":"c8a023f3-a40a-4bbe-8eb8-fbfbf1efc8de","Type":"ContainerDied","Data":"998c31bb99ea0873dc31592274a3842cd9723c374233f29cc01823a1ee93aa5d"}
Jul 24 17:16:13.866680 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.866594    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9300b6a3-646cd48f68-cb5k9"
Jul 24 17:16:13.866680 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.866642    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9300b6a3-646cd48f68-cb5k9" event={"ID":"c8a023f3-a40a-4bbe-8eb8-fbfbf1efc8de","Type":"ContainerDied","Data":"613e72503718516e5ccae514faf73ca99a979aadcba104338f1dc452fa978103"}
Jul 24 17:16:13.868111 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.868082    2563 generic.go:356] "Generic (PLEG): container finished" podID="bae67e99-d96b-40d9-9a2e-53f843a13c6b" containerID="470bf7e93906dcd14319763699e22f92899e9087901d48838b7bb82c71be2efa" exitCode=0
Jul 24 17:16:13.868246 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.868130    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9300b6a3-75f7dfdbd9-7kxcz"
Jul 24 17:16:13.868246 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.868131    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9300b6a3-75f7dfdbd9-7kxcz" event={"ID":"bae67e99-d96b-40d9-9a2e-53f843a13c6b","Type":"ContainerDied","Data":"470bf7e93906dcd14319763699e22f92899e9087901d48838b7bb82c71be2efa"}
Jul 24 17:16:13.868246 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.868189    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9300b6a3-75f7dfdbd9-7kxcz" event={"ID":"bae67e99-d96b-40d9-9a2e-53f843a13c6b","Type":"ContainerDied","Data":"876f9dfbc765a50ee3fe5525b20e38d27aad25c3249b59f4aeaa03eaaf60e3c3"}
Jul 24 17:16:13.875257 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.875242    2563 scope.go:117] "RemoveContainer" containerID="2a4b4c1d9fb062981a3350f85b996413343b6622e85f45ebe26d38fd4b2ae361"
Jul 24 17:16:13.875514 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:16:13.875494    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a4b4c1d9fb062981a3350f85b996413343b6622e85f45ebe26d38fd4b2ae361\": container with ID starting with 2a4b4c1d9fb062981a3350f85b996413343b6622e85f45ebe26d38fd4b2ae361 not found: ID does not exist" containerID="2a4b4c1d9fb062981a3350f85b996413343b6622e85f45ebe26d38fd4b2ae361"
Jul 24 17:16:13.875597 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.875519    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a4b4c1d9fb062981a3350f85b996413343b6622e85f45ebe26d38fd4b2ae361"} err="failed to get container status \"2a4b4c1d9fb062981a3350f85b996413343b6622e85f45ebe26d38fd4b2ae361\": rpc error: code = NotFound desc = could not find container \"2a4b4c1d9fb062981a3350f85b996413343b6622e85f45ebe26d38fd4b2ae361\": container with ID starting with 2a4b4c1d9fb062981a3350f85b996413343b6622e85f45ebe26d38fd4b2ae361 not found: ID does not exist"
Jul 24 17:16:13.875597 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.875535    2563 scope.go:117] "RemoveContainer" containerID="998c31bb99ea0873dc31592274a3842cd9723c374233f29cc01823a1ee93aa5d"
Jul 24 17:16:13.884018 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.883999    2563 scope.go:117] "RemoveContainer" containerID="998c31bb99ea0873dc31592274a3842cd9723c374233f29cc01823a1ee93aa5d"
Jul 24 17:16:13.884288 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:16:13.884269    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"998c31bb99ea0873dc31592274a3842cd9723c374233f29cc01823a1ee93aa5d\": container with ID starting with 998c31bb99ea0873dc31592274a3842cd9723c374233f29cc01823a1ee93aa5d not found: ID does not exist" containerID="998c31bb99ea0873dc31592274a3842cd9723c374233f29cc01823a1ee93aa5d"
Jul 24 17:16:13.884347 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.884294    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"998c31bb99ea0873dc31592274a3842cd9723c374233f29cc01823a1ee93aa5d"} err="failed to get container status \"998c31bb99ea0873dc31592274a3842cd9723c374233f29cc01823a1ee93aa5d\": rpc error: code = NotFound desc = could not find container \"998c31bb99ea0873dc31592274a3842cd9723c374233f29cc01823a1ee93aa5d\": container with ID starting with 998c31bb99ea0873dc31592274a3842cd9723c374233f29cc01823a1ee93aa5d not found: ID does not exist"
Jul 24 17:16:13.884347 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.884310    2563 scope.go:117] "RemoveContainer" containerID="470bf7e93906dcd14319763699e22f92899e9087901d48838b7bb82c71be2efa"
Jul 24 17:16:13.885605 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.885583    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9300b6a3-c77d44cb-5n7hh"]
Jul 24 17:16:13.887418 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.887399    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9300b6a3-c77d44cb-5n7hh"]
Jul 24 17:16:13.892447 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.892429    2563 scope.go:117] "RemoveContainer" containerID="470bf7e93906dcd14319763699e22f92899e9087901d48838b7bb82c71be2efa"
Jul 24 17:16:13.892710 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:16:13.892693    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"470bf7e93906dcd14319763699e22f92899e9087901d48838b7bb82c71be2efa\": container with ID starting with 470bf7e93906dcd14319763699e22f92899e9087901d48838b7bb82c71be2efa not found: ID does not exist" containerID="470bf7e93906dcd14319763699e22f92899e9087901d48838b7bb82c71be2efa"
Jul 24 17:16:13.892780 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.892716    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"470bf7e93906dcd14319763699e22f92899e9087901d48838b7bb82c71be2efa"} err="failed to get container status \"470bf7e93906dcd14319763699e22f92899e9087901d48838b7bb82c71be2efa\": rpc error: code = NotFound desc = could not find container \"470bf7e93906dcd14319763699e22f92899e9087901d48838b7bb82c71be2efa\": container with ID starting with 470bf7e93906dcd14319763699e22f92899e9087901d48838b7bb82c71be2efa not found: ID does not exist"
Jul 24 17:16:13.897285 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.897262    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9300b6a3-75f7dfdbd9-7kxcz"]
Jul 24 17:16:13.899167 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.899147    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9300b6a3-75f7dfdbd9-7kxcz"]
Jul 24 17:16:13.909951 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.909921    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9300b6a3-646cd48f68-cb5k9"]
Jul 24 17:16:13.913312 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:13.913296    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9300b6a3-646cd48f68-cb5k9"]
Jul 24 17:16:14.002828 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:14.002801    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bae67e99-d96b-40d9-9a2e-53f843a13c6b" path="/var/lib/kubelet/pods/bae67e99-d96b-40d9-9a2e-53f843a13c6b/volumes"
Jul 24 17:16:14.003193 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:14.003179    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8a023f3-a40a-4bbe-8eb8-fbfbf1efc8de" path="/var/lib/kubelet/pods/c8a023f3-a40a-4bbe-8eb8-fbfbf1efc8de/volumes"
Jul 24 17:16:14.003494 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:14.003483    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1bd7192-0408-4e97-bf92-6a2a87106c55" path="/var/lib/kubelet/pods/e1bd7192-0408-4e97-bf92-6a2a87106c55/volumes"
Jul 24 17:16:17.464050 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:17.464016    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:16:18.678595 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:18.678562    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:16:18.886763 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:18.886735    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:16:19.092008 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:19.091915    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:16:19.403179 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:19.403147    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:16:19.847532 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:19.847453    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:16:21.895645 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:21.895611    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:16:21.983695 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:21.983663    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-57de5652-7467b4bbfc-kp7qq"]
Jul 24 17:16:21.984086 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:21.984073    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8a023f3-a40a-4bbe-8eb8-fbfbf1efc8de" containerName="payload-pre-processing"
Jul 24 17:16:21.984145 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:21.984088    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8a023f3-a40a-4bbe-8eb8-fbfbf1efc8de" containerName="payload-pre-processing"
Jul 24 17:16:21.984145 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:21.984097    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1bd7192-0408-4e97-bf92-6a2a87106c55" containerName="payload-processing"
Jul 24 17:16:21.984145 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:21.984102    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1bd7192-0408-4e97-bf92-6a2a87106c55" containerName="payload-processing"
Jul 24 17:16:21.984145 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:21.984111    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bae67e99-d96b-40d9-9a2e-53f843a13c6b" containerName="maas-api"
Jul 24 17:16:21.984145 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:21.984119    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bae67e99-d96b-40d9-9a2e-53f843a13c6b" containerName="maas-api"
Jul 24 17:16:21.984307 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:21.984187    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8a023f3-a40a-4bbe-8eb8-fbfbf1efc8de" containerName="payload-pre-processing"
Jul 24 17:16:21.984307 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:21.984199    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1bd7192-0408-4e97-bf92-6a2a87106c55" containerName="payload-processing"
Jul 24 17:16:21.984307 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:21.984210    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bae67e99-d96b-40d9-9a2e-53f843a13c6b" containerName="maas-api"
Jul 24 17:16:21.988980 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:21.988957    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-57de5652-7467b4bbfc-kp7qq"
Jul 24 17:16:21.991678 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:21.991658    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-57de5652-serving-cert\""
Jul 24 17:16:21.993321 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:21.993300    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:16:22.004612 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:22.004590    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-57de5652-7467b4bbfc-kp7qq"]
Jul 24 17:16:22.072394 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:22.072357    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:16:22.123474 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:22.123437    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63e0d73a-913d-4318-b58d-3833a760ce9e-maas-api-tls\") pod \"maas-api-e2e-derive-57de5652-7467b4bbfc-kp7qq\" (UID: \"63e0d73a-913d-4318-b58d-3833a760ce9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-57de5652-7467b4bbfc-kp7qq"
Jul 24 17:16:22.123679 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:22.123502    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63e0d73a-913d-4318-b58d-3833a760ce9e-openshift-service-ca\") pod \"maas-api-e2e-derive-57de5652-7467b4bbfc-kp7qq\" (UID: \"63e0d73a-913d-4318-b58d-3833a760ce9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-57de5652-7467b4bbfc-kp7qq"
Jul 24 17:16:22.123679 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:22.123584    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdcrl\" (UniqueName: \"kubernetes.io/projected/63e0d73a-913d-4318-b58d-3833a760ce9e-kube-api-access-wdcrl\") pod \"maas-api-e2e-derive-57de5652-7467b4bbfc-kp7qq\" (UID: \"63e0d73a-913d-4318-b58d-3833a760ce9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-57de5652-7467b4bbfc-kp7qq"
Jul 24 17:16:22.202851 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:22.202772    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:16:22.224980 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:22.224914    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63e0d73a-913d-4318-b58d-3833a760ce9e-openshift-service-ca\") pod \"maas-api-e2e-derive-57de5652-7467b4bbfc-kp7qq\" (UID: \"63e0d73a-913d-4318-b58d-3833a760ce9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-57de5652-7467b4bbfc-kp7qq"
Jul 24 17:16:22.225111 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:22.225000    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdcrl\" (UniqueName: \"kubernetes.io/projected/63e0d73a-913d-4318-b58d-3833a760ce9e-kube-api-access-wdcrl\") pod \"maas-api-e2e-derive-57de5652-7467b4bbfc-kp7qq\" (UID: \"63e0d73a-913d-4318-b58d-3833a760ce9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-57de5652-7467b4bbfc-kp7qq"
Jul 24 17:16:22.225111 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:22.225050    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63e0d73a-913d-4318-b58d-3833a760ce9e-maas-api-tls\") pod \"maas-api-e2e-derive-57de5652-7467b4bbfc-kp7qq\" (UID: \"63e0d73a-913d-4318-b58d-3833a760ce9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-57de5652-7467b4bbfc-kp7qq"
Jul 24 17:16:22.225547 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:22.225528    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63e0d73a-913d-4318-b58d-3833a760ce9e-openshift-service-ca\") pod \"maas-api-e2e-derive-57de5652-7467b4bbfc-kp7qq\" (UID: \"63e0d73a-913d-4318-b58d-3833a760ce9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-57de5652-7467b4bbfc-kp7qq"
Jul 24 17:16:22.227429 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:22.227408    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63e0d73a-913d-4318-b58d-3833a760ce9e-maas-api-tls\") pod \"maas-api-e2e-derive-57de5652-7467b4bbfc-kp7qq\" (UID: \"63e0d73a-913d-4318-b58d-3833a760ce9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-57de5652-7467b4bbfc-kp7qq"
Jul 24 17:16:22.233050 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:22.233028    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdcrl\" (UniqueName: \"kubernetes.io/projected/63e0d73a-913d-4318-b58d-3833a760ce9e-kube-api-access-wdcrl\") pod \"maas-api-e2e-derive-57de5652-7467b4bbfc-kp7qq\" (UID: \"63e0d73a-913d-4318-b58d-3833a760ce9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-57de5652-7467b4bbfc-kp7qq"
Jul 24 17:16:22.301048 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:22.301010    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-57de5652-7467b4bbfc-kp7qq"
Jul 24 17:16:22.465955 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:22.465861    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-57de5652-7467b4bbfc-kp7qq"]
Jul 24 17:16:22.469505 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:16:22.469476    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63e0d73a_913d_4318_b58d_3833a760ce9e.slice/crio-9243f01802978b4eca2b2572be5516086681c185a05402baea7a75e3ed5011dd WatchSource:0}: Error finding container 9243f01802978b4eca2b2572be5516086681c185a05402baea7a75e3ed5011dd: Status 404 returned error can't find the container with id 9243f01802978b4eca2b2572be5516086681c185a05402baea7a75e3ed5011dd
Jul 24 17:16:22.701175 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:22.701141    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:16:22.796554 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:22.796474    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-57de5652-7599944ff6-kt85t"]
Jul 24 17:16:22.797667 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:22.797625    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:16:22.800093 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:22.800073    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-57de5652-7599944ff6-kt85t"
Jul 24 17:16:22.802841 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:22.802821    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-57de5652-dockercfg-hn2p9\""
Jul 24 17:16:22.803057 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:22.802909    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-57de5652\""
Jul 24 17:16:22.812083 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:22.812057    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-57de5652-7599944ff6-kt85t"]
Jul 24 17:16:22.832688 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:22.832665    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-57de5652-cbd7c54bb-zz6tb"]
Jul 24 17:16:22.836223 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:22.836202    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-57de5652-cbd7c54bb-zz6tb"
Jul 24 17:16:22.847184 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:22.847162    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-57de5652-cbd7c54bb-zz6tb"]
Jul 24 17:16:22.916179 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:22.916137    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-57de5652-7467b4bbfc-kp7qq" event={"ID":"63e0d73a-913d-4318-b58d-3833a760ce9e","Type":"ContainerStarted","Data":"2a8bdade8d3eaf68ad7b6ef69775e25a0d5cb67c3df44780298155943e758775"}
Jul 24 17:16:22.916594 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:22.916206    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-57de5652-7467b4bbfc-kp7qq" event={"ID":"63e0d73a-913d-4318-b58d-3833a760ce9e","Type":"ContainerStarted","Data":"9243f01802978b4eca2b2572be5516086681c185a05402baea7a75e3ed5011dd"}
Jul 24 17:16:22.916594 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:22.916377    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-57de5652-7467b4bbfc-kp7qq"
Jul 24 17:16:22.931523 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:22.931476    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281f2a98-1cd1-48fd-88fe-a681917f4259-plugins-config-volume\") pod \"payload-processing-e2e-derive-57de5652-cbd7c54bb-zz6tb\" (UID: \"281f2a98-1cd1-48fd-88fe-a681917f4259\") " pod="openshift-ingress/payload-processing-e2e-derive-57de5652-cbd7c54bb-zz6tb"
Jul 24 17:16:22.931523 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:22.931522    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt74t\" (UniqueName: \"kubernetes.io/projected/281f2a98-1cd1-48fd-88fe-a681917f4259-kube-api-access-bt74t\") pod \"payload-processing-e2e-derive-57de5652-cbd7c54bb-zz6tb\" (UID: \"281f2a98-1cd1-48fd-88fe-a681917f4259\") " pod="openshift-ingress/payload-processing-e2e-derive-57de5652-cbd7c54bb-zz6tb"
Jul 24 17:16:22.931723 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:22.931552    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cf29487-30e3-44fe-885b-a1d818c187d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-57de5652-7599944ff6-kt85t\" (UID: \"8cf29487-30e3-44fe-885b-a1d818c187d3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-57de5652-7599944ff6-kt85t"
Jul 24 17:16:22.931723 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:22.931669    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzngg\" (UniqueName: \"kubernetes.io/projected/8cf29487-30e3-44fe-885b-a1d818c187d3-kube-api-access-dzngg\") pod \"payload-pre-processing-e2e-derive-57de5652-7599944ff6-kt85t\" (UID: \"8cf29487-30e3-44fe-885b-a1d818c187d3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-57de5652-7599944ff6-kt85t"
Jul 24 17:16:22.932694 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:22.932641    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-57de5652-7467b4bbfc-kp7qq" podStartSLOduration=1.574724768 podStartE2EDuration="1.932624684s" totalImagesPullingTime="357.899916ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:16:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:16:22.931577018 +0000 UTC m=+2005.573540960" watchObservedRunningTime="2026-07-24 17:16:22.932624684 +0000 UTC m=+2005.574588568"
Jul 24 17:16:23.032743 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:23.032695    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281f2a98-1cd1-48fd-88fe-a681917f4259-plugins-config-volume\") pod \"payload-processing-e2e-derive-57de5652-cbd7c54bb-zz6tb\" (UID: \"281f2a98-1cd1-48fd-88fe-a681917f4259\") " pod="openshift-ingress/payload-processing-e2e-derive-57de5652-cbd7c54bb-zz6tb"
Jul 24 17:16:23.032971 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:23.032750    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt74t\" (UniqueName: \"kubernetes.io/projected/281f2a98-1cd1-48fd-88fe-a681917f4259-kube-api-access-bt74t\") pod \"payload-processing-e2e-derive-57de5652-cbd7c54bb-zz6tb\" (UID: \"281f2a98-1cd1-48fd-88fe-a681917f4259\") " pod="openshift-ingress/payload-processing-e2e-derive-57de5652-cbd7c54bb-zz6tb"
Jul 24 17:16:23.032971 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:23.032779    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cf29487-30e3-44fe-885b-a1d818c187d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-57de5652-7599944ff6-kt85t\" (UID: \"8cf29487-30e3-44fe-885b-a1d818c187d3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-57de5652-7599944ff6-kt85t"
Jul 24 17:16:23.032971 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:23.032855    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzngg\" (UniqueName: \"kubernetes.io/projected/8cf29487-30e3-44fe-885b-a1d818c187d3-kube-api-access-dzngg\") pod \"payload-pre-processing-e2e-derive-57de5652-7599944ff6-kt85t\" (UID: \"8cf29487-30e3-44fe-885b-a1d818c187d3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-57de5652-7599944ff6-kt85t"
Jul 24 17:16:23.033543 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:23.033520    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281f2a98-1cd1-48fd-88fe-a681917f4259-plugins-config-volume\") pod \"payload-processing-e2e-derive-57de5652-cbd7c54bb-zz6tb\" (UID: \"281f2a98-1cd1-48fd-88fe-a681917f4259\") " pod="openshift-ingress/payload-processing-e2e-derive-57de5652-cbd7c54bb-zz6tb"
Jul 24 17:16:23.033621 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:23.033527    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cf29487-30e3-44fe-885b-a1d818c187d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-57de5652-7599944ff6-kt85t\" (UID: \"8cf29487-30e3-44fe-885b-a1d818c187d3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-57de5652-7599944ff6-kt85t"
Jul 24 17:16:23.041193 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:23.041170    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzngg\" (UniqueName: \"kubernetes.io/projected/8cf29487-30e3-44fe-885b-a1d818c187d3-kube-api-access-dzngg\") pod \"payload-pre-processing-e2e-derive-57de5652-7599944ff6-kt85t\" (UID: \"8cf29487-30e3-44fe-885b-a1d818c187d3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-57de5652-7599944ff6-kt85t"
Jul 24 17:16:23.041417 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:23.041396    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt74t\" (UniqueName: \"kubernetes.io/projected/281f2a98-1cd1-48fd-88fe-a681917f4259-kube-api-access-bt74t\") pod \"payload-processing-e2e-derive-57de5652-cbd7c54bb-zz6tb\" (UID: \"281f2a98-1cd1-48fd-88fe-a681917f4259\") " pod="openshift-ingress/payload-processing-e2e-derive-57de5652-cbd7c54bb-zz6tb"
Jul 24 17:16:23.111494 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:23.111410    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-57de5652-7599944ff6-kt85t"
Jul 24 17:16:23.147345 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:23.147316    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-57de5652-cbd7c54bb-zz6tb"
Jul 24 17:16:23.291155 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:23.291119    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-57de5652-7599944ff6-kt85t"]
Jul 24 17:16:23.292607 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:16:23.292578    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cf29487_30e3_44fe_885b_a1d818c187d3.slice/crio-a79d716166c57c846bc912582cb3d4b0ad49749b06188d7b10f90b6fa2a13ccf WatchSource:0}: Error finding container a79d716166c57c846bc912582cb3d4b0ad49749b06188d7b10f90b6fa2a13ccf: Status 404 returned error can't find the container with id a79d716166c57c846bc912582cb3d4b0ad49749b06188d7b10f90b6fa2a13ccf
Jul 24 17:16:23.327754 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:23.327724    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-57de5652-cbd7c54bb-zz6tb"]
Jul 24 17:16:23.338281 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:16:23.338250    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod281f2a98_1cd1_48fd_88fe_a681917f4259.slice/crio-63fc52206e45ad5e08dbb6850b5477a5dbdeb372c20df3270b667bf2a45d05de WatchSource:0}: Error finding container 63fc52206e45ad5e08dbb6850b5477a5dbdeb372c20df3270b667bf2a45d05de: Status 404 returned error can't find the container with id 63fc52206e45ad5e08dbb6850b5477a5dbdeb372c20df3270b667bf2a45d05de
Jul 24 17:16:23.923015 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:23.922981    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-57de5652-7599944ff6-kt85t" event={"ID":"8cf29487-30e3-44fe-885b-a1d818c187d3","Type":"ContainerStarted","Data":"e36092b379238ed8acd360087bb54188a470ce69a29698679f755de6937eacd7"}
Jul 24 17:16:23.923500 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:23.923037    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-57de5652-7599944ff6-kt85t" event={"ID":"8cf29487-30e3-44fe-885b-a1d818c187d3","Type":"ContainerStarted","Data":"a79d716166c57c846bc912582cb3d4b0ad49749b06188d7b10f90b6fa2a13ccf"}
Jul 24 17:16:23.923500 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:23.923097    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-57de5652-7599944ff6-kt85t"
Jul 24 17:16:23.924748 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:23.924722    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-57de5652-cbd7c54bb-zz6tb" event={"ID":"281f2a98-1cd1-48fd-88fe-a681917f4259","Type":"ContainerStarted","Data":"baac4a76c5f49b30b844dd5efc612ad97d9a6d502ff4a97818a572dafc10be10"}
Jul 24 17:16:23.924866 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:23.924773    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-57de5652-cbd7c54bb-zz6tb" event={"ID":"281f2a98-1cd1-48fd-88fe-a681917f4259","Type":"ContainerStarted","Data":"63fc52206e45ad5e08dbb6850b5477a5dbdeb372c20df3270b667bf2a45d05de"}
Jul 24 17:16:23.924866 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:23.924837    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-57de5652-cbd7c54bb-zz6tb"
Jul 24 17:16:23.938158 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:23.938116    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-57de5652-7599944ff6-kt85t" podStartSLOduration=1.938104167 podStartE2EDuration="1.938104167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:16:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:16:23.937271112 +0000 UTC m=+2006.579235006" watchObservedRunningTime="2026-07-24 17:16:23.938104167 +0000 UTC m=+2006.580068052"
Jul 24 17:16:23.953132 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:23.953091    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-57de5652-cbd7c54bb-zz6tb" podStartSLOduration=1.953080254 podStartE2EDuration="1.953080254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:16:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:16:23.952661901 +0000 UTC m=+2006.594625802" watchObservedRunningTime="2026-07-24 17:16:23.953080254 +0000 UTC m=+2006.595044139"
Jul 24 17:16:28.930293 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:28.930265    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-57de5652-7467b4bbfc-kp7qq"
Jul 24 17:16:34.932081 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:34.932048    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-57de5652-7599944ff6-kt85t"
Jul 24 17:16:34.932525 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:34.932120    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-57de5652-cbd7c54bb-zz6tb"
Jul 24 17:16:57.659920 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:57.659875    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-57de5652-7467b4bbfc-kp7qq"]
Jul 24 17:16:57.660432 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:57.660165    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-57de5652-7467b4bbfc-kp7qq" podUID="63e0d73a-913d-4318-b58d-3833a760ce9e" containerName="maas-api" containerID="cri-o://2a8bdade8d3eaf68ad7b6ef69775e25a0d5cb67c3df44780298155943e758775" gracePeriod=30
Jul 24 17:16:57.667739 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:57.667708    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:16:57.700804 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:57.700761    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-57de5652-cbd7c54bb-zz6tb"]
Jul 24 17:16:57.701076 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:57.701050    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-57de5652-cbd7c54bb-zz6tb" podUID="281f2a98-1cd1-48fd-88fe-a681917f4259" containerName="payload-processing" containerID="cri-o://baac4a76c5f49b30b844dd5efc612ad97d9a6d502ff4a97818a572dafc10be10" gracePeriod=30
Jul 24 17:16:57.708729 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:57.708689    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-57de5652-7599944ff6-kt85t"]
Jul 24 17:16:57.709079 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:57.709043    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-57de5652-7599944ff6-kt85t" podUID="8cf29487-30e3-44fe-885b-a1d818c187d3" containerName="payload-pre-processing" containerID="cri-o://e36092b379238ed8acd360087bb54188a470ce69a29698679f755de6937eacd7" gracePeriod=30
Jul 24 17:16:57.740735 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:57.740702    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:16:57.798007 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:57.797979    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:16:57.971597 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:57.971572    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-57de5652-7467b4bbfc-kp7qq"
Jul 24 17:16:57.973965 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:57.973932    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-57de5652-cbd7c54bb-zz6tb"
Jul 24 17:16:58.012258 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.012228    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-57de5652-7599944ff6-kt85t"
Jul 24 17:16:58.041818 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.041800    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cf29487-30e3-44fe-885b-a1d818c187d3-plugins-config-volume\") pod \"8cf29487-30e3-44fe-885b-a1d818c187d3\" (UID: \"8cf29487-30e3-44fe-885b-a1d818c187d3\") "
Jul 24 17:16:58.041902 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.041834    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzngg\" (UniqueName: \"kubernetes.io/projected/8cf29487-30e3-44fe-885b-a1d818c187d3-kube-api-access-dzngg\") pod \"8cf29487-30e3-44fe-885b-a1d818c187d3\" (UID: \"8cf29487-30e3-44fe-885b-a1d818c187d3\") "
Jul 24 17:16:58.041902 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.041868    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63e0d73a-913d-4318-b58d-3833a760ce9e-maas-api-tls\") pod \"63e0d73a-913d-4318-b58d-3833a760ce9e\" (UID: \"63e0d73a-913d-4318-b58d-3833a760ce9e\") "
Jul 24 17:16:58.041902 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.041885    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdcrl\" (UniqueName: \"kubernetes.io/projected/63e0d73a-913d-4318-b58d-3833a760ce9e-kube-api-access-wdcrl\") pod \"63e0d73a-913d-4318-b58d-3833a760ce9e\" (UID: \"63e0d73a-913d-4318-b58d-3833a760ce9e\") "
Jul 24 17:16:58.042182 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.041923    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63e0d73a-913d-4318-b58d-3833a760ce9e-openshift-service-ca\") pod \"63e0d73a-913d-4318-b58d-3833a760ce9e\" (UID: \"63e0d73a-913d-4318-b58d-3833a760ce9e\") "
Jul 24 17:16:58.042182 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.042013    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281f2a98-1cd1-48fd-88fe-a681917f4259-plugins-config-volume\") pod \"281f2a98-1cd1-48fd-88fe-a681917f4259\" (UID: \"281f2a98-1cd1-48fd-88fe-a681917f4259\") "
Jul 24 17:16:58.042182 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.042056    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bt74t\" (UniqueName: \"kubernetes.io/projected/281f2a98-1cd1-48fd-88fe-a681917f4259-kube-api-access-bt74t\") pod \"281f2a98-1cd1-48fd-88fe-a681917f4259\" (UID: \"281f2a98-1cd1-48fd-88fe-a681917f4259\") "
Jul 24 17:16:58.042349 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.042259    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cf29487-30e3-44fe-885b-a1d818c187d3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8cf29487-30e3-44fe-885b-a1d818c187d3" (UID: "8cf29487-30e3-44fe-885b-a1d818c187d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:16:58.042409 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.042363    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cf29487-30e3-44fe-885b-a1d818c187d3-plugins-config-volume\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:16:58.042665 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.042632    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/281f2a98-1cd1-48fd-88fe-a681917f4259-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "281f2a98-1cd1-48fd-88fe-a681917f4259" (UID: "281f2a98-1cd1-48fd-88fe-a681917f4259"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:16:58.042892 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.042857    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63e0d73a-913d-4318-b58d-3833a760ce9e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "63e0d73a-913d-4318-b58d-3833a760ce9e" (UID: "63e0d73a-913d-4318-b58d-3833a760ce9e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:16:58.044373 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.044346    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cf29487-30e3-44fe-885b-a1d818c187d3-kube-api-access-dzngg" (OuterVolumeSpecName: "kube-api-access-dzngg") pod "8cf29487-30e3-44fe-885b-a1d818c187d3" (UID: "8cf29487-30e3-44fe-885b-a1d818c187d3"). InnerVolumeSpecName "kube-api-access-dzngg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:16:58.044597 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.044573    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/281f2a98-1cd1-48fd-88fe-a681917f4259-kube-api-access-bt74t" (OuterVolumeSpecName: "kube-api-access-bt74t") pod "281f2a98-1cd1-48fd-88fe-a681917f4259" (UID: "281f2a98-1cd1-48fd-88fe-a681917f4259"). InnerVolumeSpecName "kube-api-access-bt74t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:16:58.045445 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.045426    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63e0d73a-913d-4318-b58d-3833a760ce9e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "63e0d73a-913d-4318-b58d-3833a760ce9e" (UID: "63e0d73a-913d-4318-b58d-3833a760ce9e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:16:58.045445 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.045417    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63e0d73a-913d-4318-b58d-3833a760ce9e-kube-api-access-wdcrl" (OuterVolumeSpecName: "kube-api-access-wdcrl") pod "63e0d73a-913d-4318-b58d-3833a760ce9e" (UID: "63e0d73a-913d-4318-b58d-3833a760ce9e"). InnerVolumeSpecName "kube-api-access-wdcrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:16:58.094444 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.094397    2563 generic.go:356] "Generic (PLEG): container finished" podID="63e0d73a-913d-4318-b58d-3833a760ce9e" containerID="2a8bdade8d3eaf68ad7b6ef69775e25a0d5cb67c3df44780298155943e758775" exitCode=0
Jul 24 17:16:58.094444 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.094454    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-57de5652-7467b4bbfc-kp7qq"
Jul 24 17:16:58.094689 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.094488    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-57de5652-7467b4bbfc-kp7qq" event={"ID":"63e0d73a-913d-4318-b58d-3833a760ce9e","Type":"ContainerDied","Data":"2a8bdade8d3eaf68ad7b6ef69775e25a0d5cb67c3df44780298155943e758775"}
Jul 24 17:16:58.094689 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.094543    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-57de5652-7467b4bbfc-kp7qq" event={"ID":"63e0d73a-913d-4318-b58d-3833a760ce9e","Type":"ContainerDied","Data":"9243f01802978b4eca2b2572be5516086681c185a05402baea7a75e3ed5011dd"}
Jul 24 17:16:58.094689 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.094566    2563 scope.go:117] "RemoveContainer" containerID="2a8bdade8d3eaf68ad7b6ef69775e25a0d5cb67c3df44780298155943e758775"
Jul 24 17:16:58.095963 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.095913    2563 generic.go:356] "Generic (PLEG): container finished" podID="8cf29487-30e3-44fe-885b-a1d818c187d3" containerID="e36092b379238ed8acd360087bb54188a470ce69a29698679f755de6937eacd7" exitCode=0
Jul 24 17:16:58.096098 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.095984    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-57de5652-7599944ff6-kt85t"
Jul 24 17:16:58.096098 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.096001    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-57de5652-7599944ff6-kt85t" event={"ID":"8cf29487-30e3-44fe-885b-a1d818c187d3","Type":"ContainerDied","Data":"e36092b379238ed8acd360087bb54188a470ce69a29698679f755de6937eacd7"}
Jul 24 17:16:58.096098 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.096051    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-57de5652-7599944ff6-kt85t" event={"ID":"8cf29487-30e3-44fe-885b-a1d818c187d3","Type":"ContainerDied","Data":"a79d716166c57c846bc912582cb3d4b0ad49749b06188d7b10f90b6fa2a13ccf"}
Jul 24 17:16:58.097496 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.097476    2563 generic.go:356] "Generic (PLEG): container finished" podID="281f2a98-1cd1-48fd-88fe-a681917f4259" containerID="baac4a76c5f49b30b844dd5efc612ad97d9a6d502ff4a97818a572dafc10be10" exitCode=0
Jul 24 17:16:58.097601 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.097505    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-57de5652-cbd7c54bb-zz6tb" event={"ID":"281f2a98-1cd1-48fd-88fe-a681917f4259","Type":"ContainerDied","Data":"baac4a76c5f49b30b844dd5efc612ad97d9a6d502ff4a97818a572dafc10be10"}
Jul 24 17:16:58.097601 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.097540    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-57de5652-cbd7c54bb-zz6tb" event={"ID":"281f2a98-1cd1-48fd-88fe-a681917f4259","Type":"ContainerDied","Data":"63fc52206e45ad5e08dbb6850b5477a5dbdeb372c20df3270b667bf2a45d05de"}
Jul 24 17:16:58.097601 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.097550    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-57de5652-cbd7c54bb-zz6tb"
Jul 24 17:16:58.105420 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.105396    2563 scope.go:117] "RemoveContainer" containerID="2a8bdade8d3eaf68ad7b6ef69775e25a0d5cb67c3df44780298155943e758775"
Jul 24 17:16:58.106499 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:16:58.106463    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a8bdade8d3eaf68ad7b6ef69775e25a0d5cb67c3df44780298155943e758775\": container with ID starting with 2a8bdade8d3eaf68ad7b6ef69775e25a0d5cb67c3df44780298155943e758775 not found: ID does not exist" containerID="2a8bdade8d3eaf68ad7b6ef69775e25a0d5cb67c3df44780298155943e758775"
Jul 24 17:16:58.106603 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.106511    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a8bdade8d3eaf68ad7b6ef69775e25a0d5cb67c3df44780298155943e758775"} err="failed to get container status \"2a8bdade8d3eaf68ad7b6ef69775e25a0d5cb67c3df44780298155943e758775\": rpc error: code = NotFound desc = could not find container \"2a8bdade8d3eaf68ad7b6ef69775e25a0d5cb67c3df44780298155943e758775\": container with ID starting with 2a8bdade8d3eaf68ad7b6ef69775e25a0d5cb67c3df44780298155943e758775 not found: ID does not exist"
Jul 24 17:16:58.106603 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.106537    2563 scope.go:117] "RemoveContainer" containerID="e36092b379238ed8acd360087bb54188a470ce69a29698679f755de6937eacd7"
Jul 24 17:16:58.119689 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.119641    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-57de5652-7467b4bbfc-kp7qq"]
Jul 24 17:16:58.120471 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.120445    2563 scope.go:117] "RemoveContainer" containerID="e36092b379238ed8acd360087bb54188a470ce69a29698679f755de6937eacd7"
Jul 24 17:16:58.120792 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:16:58.120772    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e36092b379238ed8acd360087bb54188a470ce69a29698679f755de6937eacd7\": container with ID starting with e36092b379238ed8acd360087bb54188a470ce69a29698679f755de6937eacd7 not found: ID does not exist" containerID="e36092b379238ed8acd360087bb54188a470ce69a29698679f755de6937eacd7"
Jul 24 17:16:58.120880 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.120804    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e36092b379238ed8acd360087bb54188a470ce69a29698679f755de6937eacd7"} err="failed to get container status \"e36092b379238ed8acd360087bb54188a470ce69a29698679f755de6937eacd7\": rpc error: code = NotFound desc = could not find container \"e36092b379238ed8acd360087bb54188a470ce69a29698679f755de6937eacd7\": container with ID starting with e36092b379238ed8acd360087bb54188a470ce69a29698679f755de6937eacd7 not found: ID does not exist"
Jul 24 17:16:58.120880 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.120830    2563 scope.go:117] "RemoveContainer" containerID="baac4a76c5f49b30b844dd5efc612ad97d9a6d502ff4a97818a572dafc10be10"
Jul 24 17:16:58.121757 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.121739    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-57de5652-7467b4bbfc-kp7qq"]
Jul 24 17:16:58.128800 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.128781    2563 scope.go:117] "RemoveContainer" containerID="baac4a76c5f49b30b844dd5efc612ad97d9a6d502ff4a97818a572dafc10be10"
Jul 24 17:16:58.129072 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:16:58.129055    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baac4a76c5f49b30b844dd5efc612ad97d9a6d502ff4a97818a572dafc10be10\": container with ID starting with baac4a76c5f49b30b844dd5efc612ad97d9a6d502ff4a97818a572dafc10be10 not found: ID does not exist" containerID="baac4a76c5f49b30b844dd5efc612ad97d9a6d502ff4a97818a572dafc10be10"
Jul 24 17:16:58.129139 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.129078    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baac4a76c5f49b30b844dd5efc612ad97d9a6d502ff4a97818a572dafc10be10"} err="failed to get container status \"baac4a76c5f49b30b844dd5efc612ad97d9a6d502ff4a97818a572dafc10be10\": rpc error: code = NotFound desc = could not find container \"baac4a76c5f49b30b844dd5efc612ad97d9a6d502ff4a97818a572dafc10be10\": container with ID starting with baac4a76c5f49b30b844dd5efc612ad97d9a6d502ff4a97818a572dafc10be10 not found: ID does not exist"
Jul 24 17:16:58.132041 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.132019    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-57de5652-cbd7c54bb-zz6tb"]
Jul 24 17:16:58.135531 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.135510    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-57de5652-cbd7c54bb-zz6tb"]
Jul 24 17:16:58.143242 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.143219    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzngg\" (UniqueName: \"kubernetes.io/projected/8cf29487-30e3-44fe-885b-a1d818c187d3-kube-api-access-dzngg\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:16:58.143320 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.143244    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63e0d73a-913d-4318-b58d-3833a760ce9e-maas-api-tls\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:16:58.143320 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.143255    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdcrl\" (UniqueName: \"kubernetes.io/projected/63e0d73a-913d-4318-b58d-3833a760ce9e-kube-api-access-wdcrl\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:16:58.143320 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.143264    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63e0d73a-913d-4318-b58d-3833a760ce9e-openshift-service-ca\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:16:58.143320 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.143273    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281f2a98-1cd1-48fd-88fe-a681917f4259-plugins-config-volume\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:16:58.143320 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.143280    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt74t\" (UniqueName: \"kubernetes.io/projected/281f2a98-1cd1-48fd-88fe-a681917f4259-kube-api-access-bt74t\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:16:58.146863 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.146838    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-57de5652-7599944ff6-kt85t"]
Jul 24 17:16:58.151122 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:16:58.151100    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-57de5652-7599944ff6-kt85t"]
Jul 24 17:17:00.002916 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:17:00.002880    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="281f2a98-1cd1-48fd-88fe-a681917f4259" path="/var/lib/kubelet/pods/281f2a98-1cd1-48fd-88fe-a681917f4259/volumes"
Jul 24 17:17:00.003395 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:17:00.003327    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63e0d73a-913d-4318-b58d-3833a760ce9e" path="/var/lib/kubelet/pods/63e0d73a-913d-4318-b58d-3833a760ce9e/volumes"
Jul 24 17:17:00.003656 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:17:00.003642    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cf29487-30e3-44fe-885b-a1d818c187d3" path="/var/lib/kubelet/pods/8cf29487-30e3-44fe-885b-a1d818c187d3/volumes"
Jul 24 17:17:01.854545 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:17:01.854508    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:17:02.006320 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:17:02.006290    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:17:02.235335 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:17:02.235297    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:17:02.715118 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:17:02.715085    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:17:02.812708 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:17:02.812667    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:17:02.912228 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:17:02.912183    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:17:04.331765 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:17:04.331729    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:17:10.513880 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:17:10.513846    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:17:10.620032 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:17:10.620003    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:17:10.715021 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:17:10.714991    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:17:10.815107 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:17:10.815029    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:17:11.223572 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:17:11.223542    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:17:34.583406 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:17:34.583313    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:17:37.167703 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:17:37.167671    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:17:43.432588 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:17:43.432545    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:17:43.541502 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:17:43.541451    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:17:43.634211 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:17:43.634179    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:17:50.216661 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:17:50.216620    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:17:54.924089 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:17:54.924057    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:17:54.976130 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:17:54.976096    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:17:58.114571 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:17:58.114548    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ddklx_fb5a9838-1e05-4f86-b46c-760bd7aaa038/console-operator/1.log"
Jul 24 17:17:58.121483 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:17:58.121460    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ddklx_fb5a9838-1e05-4f86-b46c-760bd7aaa038/console-operator/1.log"
Jul 24 17:17:59.515841 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:17:59.515809    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:05.482785 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:05.482746    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:05.679664 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:05.679631    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:05.806154 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:05.806071    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:16.245543 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:16.245507    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:17.049409 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:17.049374    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:23.177159 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:23.177128    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:23.285434 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:23.285397    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:23.382232 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:23.382200    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:23.483099 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:23.483015    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:24.302604 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:24.302566    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:24.403864 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:24.403830    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:24.523630 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:24.523599    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:26.177067 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:26.177030    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:32.387277 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:32.387235    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:32.496873 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:32.496844    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:33.108347 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:33.108310    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:39.366481 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:39.366443    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:39.429625 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:39.429595    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:39.608142 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:39.608101    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:39.917248 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:39.917216    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:40.340464 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:40.340374    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:43.433829 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:43.433791    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:43.505356 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:43.505330    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e30ccfde-7c86988c55-8r8zh"]
Jul 24 17:18:43.505688 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:43.505677    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63e0d73a-913d-4318-b58d-3833a760ce9e" containerName="maas-api"
Jul 24 17:18:43.505733 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:43.505689    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="63e0d73a-913d-4318-b58d-3833a760ce9e" containerName="maas-api"
Jul 24 17:18:43.505733 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:43.505704    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cf29487-30e3-44fe-885b-a1d818c187d3" containerName="payload-pre-processing"
Jul 24 17:18:43.505733 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:43.505710    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cf29487-30e3-44fe-885b-a1d818c187d3" containerName="payload-pre-processing"
Jul 24 17:18:43.505733 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:43.505721    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="281f2a98-1cd1-48fd-88fe-a681917f4259" containerName="payload-processing"
Jul 24 17:18:43.505733 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:43.505726    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="281f2a98-1cd1-48fd-88fe-a681917f4259" containerName="payload-processing"
Jul 24 17:18:43.505976 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:43.505784    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="281f2a98-1cd1-48fd-88fe-a681917f4259" containerName="payload-processing"
Jul 24 17:18:43.505976 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:43.505792    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="63e0d73a-913d-4318-b58d-3833a760ce9e" containerName="maas-api"
Jul 24 17:18:43.505976 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:43.505806    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cf29487-30e3-44fe-885b-a1d818c187d3" containerName="payload-pre-processing"
Jul 24 17:18:43.509179 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:43.509159    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30ccfde-7c86988c55-8r8zh"
Jul 24 17:18:43.511456 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:43.511426    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e30ccfde-serving-cert\""
Jul 24 17:18:43.511566 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:43.511456    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:43.521236 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:43.521211    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e30ccfde-7c86988c55-8r8zh"]
Jul 24 17:18:43.535510 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:43.535478    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58gdd\" (UniqueName: \"kubernetes.io/projected/bc1b62d2-88b3-4679-99ad-546ad19bb589-kube-api-access-58gdd\") pod \"maas-api-e2e-mt-e30ccfde-7c86988c55-8r8zh\" (UID: \"bc1b62d2-88b3-4679-99ad-546ad19bb589\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30ccfde-7c86988c55-8r8zh"
Jul 24 17:18:43.535617 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:43.535539    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc1b62d2-88b3-4679-99ad-546ad19bb589-maas-api-tls\") pod \"maas-api-e2e-mt-e30ccfde-7c86988c55-8r8zh\" (UID: \"bc1b62d2-88b3-4679-99ad-546ad19bb589\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30ccfde-7c86988c55-8r8zh"
Jul 24 17:18:43.535617 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:43.535595    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc1b62d2-88b3-4679-99ad-546ad19bb589-openshift-service-ca\") pod \"maas-api-e2e-mt-e30ccfde-7c86988c55-8r8zh\" (UID: \"bc1b62d2-88b3-4679-99ad-546ad19bb589\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30ccfde-7c86988c55-8r8zh"
Jul 24 17:18:43.582732 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:43.582702    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:43.636433 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:43.636402    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc1b62d2-88b3-4679-99ad-546ad19bb589-openshift-service-ca\") pod \"maas-api-e2e-mt-e30ccfde-7c86988c55-8r8zh\" (UID: \"bc1b62d2-88b3-4679-99ad-546ad19bb589\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30ccfde-7c86988c55-8r8zh"
Jul 24 17:18:43.636611 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:43.636463    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58gdd\" (UniqueName: \"kubernetes.io/projected/bc1b62d2-88b3-4679-99ad-546ad19bb589-kube-api-access-58gdd\") pod \"maas-api-e2e-mt-e30ccfde-7c86988c55-8r8zh\" (UID: \"bc1b62d2-88b3-4679-99ad-546ad19bb589\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30ccfde-7c86988c55-8r8zh"
Jul 24 17:18:43.636611 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:43.636490    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc1b62d2-88b3-4679-99ad-546ad19bb589-maas-api-tls\") pod \"maas-api-e2e-mt-e30ccfde-7c86988c55-8r8zh\" (UID: \"bc1b62d2-88b3-4679-99ad-546ad19bb589\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30ccfde-7c86988c55-8r8zh"
Jul 24 17:18:43.636611 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:18:43.636577    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-e30ccfde-serving-cert: secret "maas-api-e2e-mt-e30ccfde-serving-cert" not found
Jul 24 17:18:43.636795 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:18:43.636634    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc1b62d2-88b3-4679-99ad-546ad19bb589-maas-api-tls podName:bc1b62d2-88b3-4679-99ad-546ad19bb589 nodeName:}" failed. No retries permitted until 2026-07-24 17:18:44.136619191 +0000 UTC m=+2146.778583053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bc1b62d2-88b3-4679-99ad-546ad19bb589-maas-api-tls") pod "maas-api-e2e-mt-e30ccfde-7c86988c55-8r8zh" (UID: "bc1b62d2-88b3-4679-99ad-546ad19bb589") : secret "maas-api-e2e-mt-e30ccfde-serving-cert" not found
Jul 24 17:18:43.637144 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:43.637122    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc1b62d2-88b3-4679-99ad-546ad19bb589-openshift-service-ca\") pod \"maas-api-e2e-mt-e30ccfde-7c86988c55-8r8zh\" (UID: \"bc1b62d2-88b3-4679-99ad-546ad19bb589\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30ccfde-7c86988c55-8r8zh"
Jul 24 17:18:43.644979 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:43.644954    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58gdd\" (UniqueName: \"kubernetes.io/projected/bc1b62d2-88b3-4679-99ad-546ad19bb589-kube-api-access-58gdd\") pod \"maas-api-e2e-mt-e30ccfde-7c86988c55-8r8zh\" (UID: \"bc1b62d2-88b3-4679-99ad-546ad19bb589\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30ccfde-7c86988c55-8r8zh"
Jul 24 17:18:43.693019 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:43.692920    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:44.140447 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:44.140418    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc1b62d2-88b3-4679-99ad-546ad19bb589-maas-api-tls\") pod \"maas-api-e2e-mt-e30ccfde-7c86988c55-8r8zh\" (UID: \"bc1b62d2-88b3-4679-99ad-546ad19bb589\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30ccfde-7c86988c55-8r8zh"
Jul 24 17:18:44.143103 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:44.143080    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc1b62d2-88b3-4679-99ad-546ad19bb589-maas-api-tls\") pod \"maas-api-e2e-mt-e30ccfde-7c86988c55-8r8zh\" (UID: \"bc1b62d2-88b3-4679-99ad-546ad19bb589\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30ccfde-7c86988c55-8r8zh"
Jul 24 17:18:44.202389 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:44.202354    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:44.300673 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:44.300643    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:44.323869 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:44.323836    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e30ccfde-9f66f9bbd-bjsh4"]
Jul 24 17:18:44.327478 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:44.327459    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e30ccfde-9f66f9bbd-bjsh4"
Jul 24 17:18:44.330170 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:44.330146    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e30ccfde-dockercfg-nwrbd\""
Jul 24 17:18:44.330797 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:44.330774    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e30ccfde\""
Jul 24 17:18:44.332716 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:44.332696    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e30ccfde-5b9fcc5fc8-tkqdr"]
Jul 24 17:18:44.338130 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:44.338111    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e30ccfde-9f66f9bbd-bjsh4"]
Jul 24 17:18:44.338223 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:44.338210    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e30ccfde-5b9fcc5fc8-tkqdr"
Jul 24 17:18:44.342662 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:44.342632    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d17a46f4-1ca4-4674-8879-b82ff3e7df10-plugins-config-volume\") pod \"payload-processing-e2e-mt-e30ccfde-5b9fcc5fc8-tkqdr\" (UID: \"d17a46f4-1ca4-4674-8879-b82ff3e7df10\") " pod="openshift-ingress/payload-processing-e2e-mt-e30ccfde-5b9fcc5fc8-tkqdr"
Jul 24 17:18:44.342777 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:44.342710    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13dcb0d8-b891-4f9c-aedd-4dfe1cdcc6a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e30ccfde-9f66f9bbd-bjsh4\" (UID: \"13dcb0d8-b891-4f9c-aedd-4dfe1cdcc6a3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e30ccfde-9f66f9bbd-bjsh4"
Jul 24 17:18:44.342777 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:44.342743    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgd92\" (UniqueName: \"kubernetes.io/projected/13dcb0d8-b891-4f9c-aedd-4dfe1cdcc6a3-kube-api-access-jgd92\") pod \"payload-pre-processing-e2e-mt-e30ccfde-9f66f9bbd-bjsh4\" (UID: \"13dcb0d8-b891-4f9c-aedd-4dfe1cdcc6a3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e30ccfde-9f66f9bbd-bjsh4"
Jul 24 17:18:44.342777 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:44.342766    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzrb7\" (UniqueName: \"kubernetes.io/projected/d17a46f4-1ca4-4674-8879-b82ff3e7df10-kube-api-access-zzrb7\") pod \"payload-processing-e2e-mt-e30ccfde-5b9fcc5fc8-tkqdr\" (UID: \"d17a46f4-1ca4-4674-8879-b82ff3e7df10\") " pod="openshift-ingress/payload-processing-e2e-mt-e30ccfde-5b9fcc5fc8-tkqdr"
Jul 24 17:18:44.347646 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:44.347623    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e30ccfde-5b9fcc5fc8-tkqdr"]
Jul 24 17:18:44.421178 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:44.421094    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30ccfde-7c86988c55-8r8zh"
Jul 24 17:18:44.443803 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:44.443769    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13dcb0d8-b891-4f9c-aedd-4dfe1cdcc6a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e30ccfde-9f66f9bbd-bjsh4\" (UID: \"13dcb0d8-b891-4f9c-aedd-4dfe1cdcc6a3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e30ccfde-9f66f9bbd-bjsh4"
Jul 24 17:18:44.444260 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:44.443815    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgd92\" (UniqueName: \"kubernetes.io/projected/13dcb0d8-b891-4f9c-aedd-4dfe1cdcc6a3-kube-api-access-jgd92\") pod \"payload-pre-processing-e2e-mt-e30ccfde-9f66f9bbd-bjsh4\" (UID: \"13dcb0d8-b891-4f9c-aedd-4dfe1cdcc6a3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e30ccfde-9f66f9bbd-bjsh4"
Jul 24 17:18:44.444260 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:44.443844    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzrb7\" (UniqueName: \"kubernetes.io/projected/d17a46f4-1ca4-4674-8879-b82ff3e7df10-kube-api-access-zzrb7\") pod \"payload-processing-e2e-mt-e30ccfde-5b9fcc5fc8-tkqdr\" (UID: \"d17a46f4-1ca4-4674-8879-b82ff3e7df10\") " pod="openshift-ingress/payload-processing-e2e-mt-e30ccfde-5b9fcc5fc8-tkqdr"
Jul 24 17:18:44.444260 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:44.443966    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d17a46f4-1ca4-4674-8879-b82ff3e7df10-plugins-config-volume\") pod \"payload-processing-e2e-mt-e30ccfde-5b9fcc5fc8-tkqdr\" (UID: \"d17a46f4-1ca4-4674-8879-b82ff3e7df10\") " pod="openshift-ingress/payload-processing-e2e-mt-e30ccfde-5b9fcc5fc8-tkqdr"
Jul 24 17:18:44.444860 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:44.444837    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13dcb0d8-b891-4f9c-aedd-4dfe1cdcc6a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e30ccfde-9f66f9bbd-bjsh4\" (UID: \"13dcb0d8-b891-4f9c-aedd-4dfe1cdcc6a3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e30ccfde-9f66f9bbd-bjsh4"
Jul 24 17:18:44.444860 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:44.444849    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d17a46f4-1ca4-4674-8879-b82ff3e7df10-plugins-config-volume\") pod \"payload-processing-e2e-mt-e30ccfde-5b9fcc5fc8-tkqdr\" (UID: \"d17a46f4-1ca4-4674-8879-b82ff3e7df10\") " pod="openshift-ingress/payload-processing-e2e-mt-e30ccfde-5b9fcc5fc8-tkqdr"
Jul 24 17:18:44.461899 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:44.461869    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgd92\" (UniqueName: \"kubernetes.io/projected/13dcb0d8-b891-4f9c-aedd-4dfe1cdcc6a3-kube-api-access-jgd92\") pod \"payload-pre-processing-e2e-mt-e30ccfde-9f66f9bbd-bjsh4\" (UID: \"13dcb0d8-b891-4f9c-aedd-4dfe1cdcc6a3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e30ccfde-9f66f9bbd-bjsh4"
Jul 24 17:18:44.470581 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:44.470560    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzrb7\" (UniqueName: \"kubernetes.io/projected/d17a46f4-1ca4-4674-8879-b82ff3e7df10-kube-api-access-zzrb7\") pod \"payload-processing-e2e-mt-e30ccfde-5b9fcc5fc8-tkqdr\" (UID: \"d17a46f4-1ca4-4674-8879-b82ff3e7df10\") " pod="openshift-ingress/payload-processing-e2e-mt-e30ccfde-5b9fcc5fc8-tkqdr"
Jul 24 17:18:44.565539 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:44.565507    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e30ccfde-7c86988c55-8r8zh"]
Jul 24 17:18:44.567488 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:18:44.567461    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc1b62d2_88b3_4679_99ad_546ad19bb589.slice/crio-e76375f2aecd28748423e2f5b928c8b466ef7bf82c0b260d6879e9fc11241e7c WatchSource:0}: Error finding container e76375f2aecd28748423e2f5b928c8b466ef7bf82c0b260d6879e9fc11241e7c: Status 404 returned error can't find the container with id e76375f2aecd28748423e2f5b928c8b466ef7bf82c0b260d6879e9fc11241e7c
Jul 24 17:18:44.612417 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:44.612385    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30ccfde-7c86988c55-8r8zh" event={"ID":"bc1b62d2-88b3-4679-99ad-546ad19bb589","Type":"ContainerStarted","Data":"e76375f2aecd28748423e2f5b928c8b466ef7bf82c0b260d6879e9fc11241e7c"}
Jul 24 17:18:44.640656 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:44.640630    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e30ccfde-9f66f9bbd-bjsh4"
Jul 24 17:18:44.654697 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:44.654553    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e30ccfde-5b9fcc5fc8-tkqdr"
Jul 24 17:18:44.793243 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:44.793116    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e30ccfde-9f66f9bbd-bjsh4"]
Jul 24 17:18:44.796020 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:18:44.795992    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13dcb0d8_b891_4f9c_aedd_4dfe1cdcc6a3.slice/crio-343549290c028a908c7b76769d39809a329a49a06a4d4d97f98b9e83f076629c WatchSource:0}: Error finding container 343549290c028a908c7b76769d39809a329a49a06a4d4d97f98b9e83f076629c: Status 404 returned error can't find the container with id 343549290c028a908c7b76769d39809a329a49a06a4d4d97f98b9e83f076629c
Jul 24 17:18:44.815422 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:44.815394    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e30ccfde-5b9fcc5fc8-tkqdr"]
Jul 24 17:18:44.816872 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:18:44.816849    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd17a46f4_1ca4_4674_8879_b82ff3e7df10.slice/crio-e4246797ec76352c3c7e6f53397d1379b95ff6c54947eb59a43509dfddb1b3db WatchSource:0}: Error finding container e4246797ec76352c3c7e6f53397d1379b95ff6c54947eb59a43509dfddb1b3db: Status 404 returned error can't find the container with id e4246797ec76352c3c7e6f53397d1379b95ff6c54947eb59a43509dfddb1b3db
Jul 24 17:18:45.618801 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:45.618764    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e30ccfde-5b9fcc5fc8-tkqdr" event={"ID":"d17a46f4-1ca4-4674-8879-b82ff3e7df10","Type":"ContainerStarted","Data":"0039800a520748c40053dde20c93c98c9c8e36729af851416444002a04b3ef21"}
Jul 24 17:18:45.619259 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:45.618826    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e30ccfde-5b9fcc5fc8-tkqdr" event={"ID":"d17a46f4-1ca4-4674-8879-b82ff3e7df10","Type":"ContainerStarted","Data":"e4246797ec76352c3c7e6f53397d1379b95ff6c54947eb59a43509dfddb1b3db"}
Jul 24 17:18:45.619259 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:45.618857    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-e30ccfde-5b9fcc5fc8-tkqdr"
Jul 24 17:18:45.620335 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:45.620309    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e30ccfde-9f66f9bbd-bjsh4" event={"ID":"13dcb0d8-b891-4f9c-aedd-4dfe1cdcc6a3","Type":"ContainerStarted","Data":"47469a65a7e8a545101772146b4cd0127e0f212198cba6126632a9ee61ca4476"}
Jul 24 17:18:45.620476 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:45.620359    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e30ccfde-9f66f9bbd-bjsh4" event={"ID":"13dcb0d8-b891-4f9c-aedd-4dfe1cdcc6a3","Type":"ContainerStarted","Data":"343549290c028a908c7b76769d39809a329a49a06a4d4d97f98b9e83f076629c"}
Jul 24 17:18:45.620476 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:45.620403    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e30ccfde-9f66f9bbd-bjsh4"
Jul 24 17:18:45.621839 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:45.621805    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30ccfde-7c86988c55-8r8zh" event={"ID":"bc1b62d2-88b3-4679-99ad-546ad19bb589","Type":"ContainerStarted","Data":"b77f45dd72d5d37d98ce71616d3e4397f1bef2462cd879945d5ac8c69b3e38f8"}
Jul 24 17:18:45.621980 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:45.621964    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30ccfde-7c86988c55-8r8zh"
Jul 24 17:18:45.639646 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:45.639595    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-e30ccfde-5b9fcc5fc8-tkqdr" podStartSLOduration=1.639577011 podStartE2EDuration="1.639577011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:18:45.637347948 +0000 UTC m=+2148.279311852" watchObservedRunningTime="2026-07-24 17:18:45.639577011 +0000 UTC m=+2148.281540896"
Jul 24 17:18:45.655193 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:45.655141    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-e30ccfde-9f66f9bbd-bjsh4" podStartSLOduration=1.655124625 podStartE2EDuration="1.655124625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:18:45.652423317 +0000 UTC m=+2148.294387205" watchObservedRunningTime="2026-07-24 17:18:45.655124625 +0000 UTC m=+2148.297088510"
Jul 24 17:18:45.670577 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:45.670532    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30ccfde-7c86988c55-8r8zh" podStartSLOduration=2.204783102 podStartE2EDuration="2.670517688s" totalImagesPullingTime="465.734586ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:18:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:18:45.668990047 +0000 UTC m=+2148.310953960" watchObservedRunningTime="2026-07-24 17:18:45.670517688 +0000 UTC m=+2148.312481571"
Jul 24 17:18:47.870721 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:47.870682    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:48.548016 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:48.547975    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:48.669806 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:48.669763    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:49.014287 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:49.014257    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:51.632986 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:51.632959    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30ccfde-7c86988c55-8r8zh"
Jul 24 17:18:54.677838 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:54.677802    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:54.732871 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:54.732844    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:18:56.629342 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:56.629308    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e30ccfde-9f66f9bbd-bjsh4"
Jul 24 17:18:56.629825 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:18:56.629643    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-e30ccfde-5b9fcc5fc8-tkqdr"
Jul 24 17:19:05.593188 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:05.593151    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e30ccfde-7c86988c55-8r8zh"]
Jul 24 17:19:05.593653 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:05.593370    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30ccfde-7c86988c55-8r8zh" podUID="bc1b62d2-88b3-4679-99ad-546ad19bb589" containerName="maas-api" containerID="cri-o://b77f45dd72d5d37d98ce71616d3e4397f1bef2462cd879945d5ac8c69b3e38f8" gracePeriod=30
Jul 24 17:19:05.625374 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:05.625344    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:19:05.647582 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:05.647546    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e30ccfde-5b9fcc5fc8-tkqdr"]
Jul 24 17:19:05.647931 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:05.647900    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-e30ccfde-5b9fcc5fc8-tkqdr" podUID="d17a46f4-1ca4-4674-8879-b82ff3e7df10" containerName="payload-processing" containerID="cri-o://0039800a520748c40053dde20c93c98c9c8e36729af851416444002a04b3ef21" gracePeriod=30
Jul 24 17:19:05.655991 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:05.655963    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e30ccfde-9f66f9bbd-bjsh4"]
Jul 24 17:19:05.656262 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:05.656202    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-e30ccfde-9f66f9bbd-bjsh4" podUID="13dcb0d8-b891-4f9c-aedd-4dfe1cdcc6a3" containerName="payload-pre-processing" containerID="cri-o://47469a65a7e8a545101772146b4cd0127e0f212198cba6126632a9ee61ca4476" gracePeriod=30
Jul 24 17:19:05.725502 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:05.725384    2563 generic.go:356] "Generic (PLEG): container finished" podID="bc1b62d2-88b3-4679-99ad-546ad19bb589" containerID="b77f45dd72d5d37d98ce71616d3e4397f1bef2462cd879945d5ac8c69b3e38f8" exitCode=0
Jul 24 17:19:05.725502 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:05.725424    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30ccfde-7c86988c55-8r8zh" event={"ID":"bc1b62d2-88b3-4679-99ad-546ad19bb589","Type":"ContainerDied","Data":"b77f45dd72d5d37d98ce71616d3e4397f1bef2462cd879945d5ac8c69b3e38f8"}
Jul 24 17:19:05.726349 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:05.726330    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:19:05.826727 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:05.826667    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:19:05.942568 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:05.942543    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e30ccfde-5b9fcc5fc8-tkqdr"
Jul 24 17:19:05.945584 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:05.945563    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30ccfde-7c86988c55-8r8zh"
Jul 24 17:19:05.971896 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:05.971866    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e30ccfde-9f66f9bbd-bjsh4"
Jul 24 17:19:06.038385 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.038354    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58gdd\" (UniqueName: \"kubernetes.io/projected/bc1b62d2-88b3-4679-99ad-546ad19bb589-kube-api-access-58gdd\") pod \"bc1b62d2-88b3-4679-99ad-546ad19bb589\" (UID: \"bc1b62d2-88b3-4679-99ad-546ad19bb589\") "
Jul 24 17:19:06.038599 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.038416    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc1b62d2-88b3-4679-99ad-546ad19bb589-maas-api-tls\") pod \"bc1b62d2-88b3-4679-99ad-546ad19bb589\" (UID: \"bc1b62d2-88b3-4679-99ad-546ad19bb589\") "
Jul 24 17:19:06.038599 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.038455    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzrb7\" (UniqueName: \"kubernetes.io/projected/d17a46f4-1ca4-4674-8879-b82ff3e7df10-kube-api-access-zzrb7\") pod \"d17a46f4-1ca4-4674-8879-b82ff3e7df10\" (UID: \"d17a46f4-1ca4-4674-8879-b82ff3e7df10\") "
Jul 24 17:19:06.038599 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.038472    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgd92\" (UniqueName: \"kubernetes.io/projected/13dcb0d8-b891-4f9c-aedd-4dfe1cdcc6a3-kube-api-access-jgd92\") pod \"13dcb0d8-b891-4f9c-aedd-4dfe1cdcc6a3\" (UID: \"13dcb0d8-b891-4f9c-aedd-4dfe1cdcc6a3\") "
Jul 24 17:19:06.038599 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.038490    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d17a46f4-1ca4-4674-8879-b82ff3e7df10-plugins-config-volume\") pod \"d17a46f4-1ca4-4674-8879-b82ff3e7df10\" (UID: \"d17a46f4-1ca4-4674-8879-b82ff3e7df10\") "
Jul 24 17:19:06.038599 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.038517    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc1b62d2-88b3-4679-99ad-546ad19bb589-openshift-service-ca\") pod \"bc1b62d2-88b3-4679-99ad-546ad19bb589\" (UID: \"bc1b62d2-88b3-4679-99ad-546ad19bb589\") "
Jul 24 17:19:06.038880 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.038631    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13dcb0d8-b891-4f9c-aedd-4dfe1cdcc6a3-plugins-config-volume\") pod \"13dcb0d8-b891-4f9c-aedd-4dfe1cdcc6a3\" (UID: \"13dcb0d8-b891-4f9c-aedd-4dfe1cdcc6a3\") "
Jul 24 17:19:06.038971 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.038882    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc1b62d2-88b3-4679-99ad-546ad19bb589-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bc1b62d2-88b3-4679-99ad-546ad19bb589" (UID: "bc1b62d2-88b3-4679-99ad-546ad19bb589"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:19:06.038971 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.038892    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d17a46f4-1ca4-4674-8879-b82ff3e7df10-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d17a46f4-1ca4-4674-8879-b82ff3e7df10" (UID: "d17a46f4-1ca4-4674-8879-b82ff3e7df10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:19:06.039087 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.039052    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d17a46f4-1ca4-4674-8879-b82ff3e7df10-plugins-config-volume\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:19:06.039087 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.039071    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc1b62d2-88b3-4679-99ad-546ad19bb589-openshift-service-ca\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:19:06.039300 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.039278    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13dcb0d8-b891-4f9c-aedd-4dfe1cdcc6a3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "13dcb0d8-b891-4f9c-aedd-4dfe1cdcc6a3" (UID: "13dcb0d8-b891-4f9c-aedd-4dfe1cdcc6a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:19:06.040689 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.040661    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc1b62d2-88b3-4679-99ad-546ad19bb589-kube-api-access-58gdd" (OuterVolumeSpecName: "kube-api-access-58gdd") pod "bc1b62d2-88b3-4679-99ad-546ad19bb589" (UID: "bc1b62d2-88b3-4679-99ad-546ad19bb589"). InnerVolumeSpecName "kube-api-access-58gdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:19:06.040798 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.040763    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13dcb0d8-b891-4f9c-aedd-4dfe1cdcc6a3-kube-api-access-jgd92" (OuterVolumeSpecName: "kube-api-access-jgd92") pod "13dcb0d8-b891-4f9c-aedd-4dfe1cdcc6a3" (UID: "13dcb0d8-b891-4f9c-aedd-4dfe1cdcc6a3"). InnerVolumeSpecName "kube-api-access-jgd92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:19:06.041052 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.041030    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc1b62d2-88b3-4679-99ad-546ad19bb589-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bc1b62d2-88b3-4679-99ad-546ad19bb589" (UID: "bc1b62d2-88b3-4679-99ad-546ad19bb589"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:19:06.041149 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.041051    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d17a46f4-1ca4-4674-8879-b82ff3e7df10-kube-api-access-zzrb7" (OuterVolumeSpecName: "kube-api-access-zzrb7") pod "d17a46f4-1ca4-4674-8879-b82ff3e7df10" (UID: "d17a46f4-1ca4-4674-8879-b82ff3e7df10"). InnerVolumeSpecName "kube-api-access-zzrb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:19:06.140180 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.140146    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58gdd\" (UniqueName: \"kubernetes.io/projected/bc1b62d2-88b3-4679-99ad-546ad19bb589-kube-api-access-58gdd\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:19:06.140180 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.140181    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc1b62d2-88b3-4679-99ad-546ad19bb589-maas-api-tls\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:19:06.140385 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.140194    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzrb7\" (UniqueName: \"kubernetes.io/projected/d17a46f4-1ca4-4674-8879-b82ff3e7df10-kube-api-access-zzrb7\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:19:06.140385 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.140204    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgd92\" (UniqueName: \"kubernetes.io/projected/13dcb0d8-b891-4f9c-aedd-4dfe1cdcc6a3-kube-api-access-jgd92\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:19:06.140385 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.140213    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13dcb0d8-b891-4f9c-aedd-4dfe1cdcc6a3-plugins-config-volume\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:19:06.731299 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.731273    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30ccfde-7c86988c55-8r8zh"
Jul 24 17:19:06.731299 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.731292    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30ccfde-7c86988c55-8r8zh" event={"ID":"bc1b62d2-88b3-4679-99ad-546ad19bb589","Type":"ContainerDied","Data":"e76375f2aecd28748423e2f5b928c8b466ef7bf82c0b260d6879e9fc11241e7c"}
Jul 24 17:19:06.731846 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.731357    2563 scope.go:117] "RemoveContainer" containerID="b77f45dd72d5d37d98ce71616d3e4397f1bef2462cd879945d5ac8c69b3e38f8"
Jul 24 17:19:06.732688 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.732625    2563 generic.go:356] "Generic (PLEG): container finished" podID="d17a46f4-1ca4-4674-8879-b82ff3e7df10" containerID="0039800a520748c40053dde20c93c98c9c8e36729af851416444002a04b3ef21" exitCode=0
Jul 24 17:19:06.732688 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.732678    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e30ccfde-5b9fcc5fc8-tkqdr"
Jul 24 17:19:06.732817 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.732702    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e30ccfde-5b9fcc5fc8-tkqdr" event={"ID":"d17a46f4-1ca4-4674-8879-b82ff3e7df10","Type":"ContainerDied","Data":"0039800a520748c40053dde20c93c98c9c8e36729af851416444002a04b3ef21"}
Jul 24 17:19:06.732817 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.732742    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e30ccfde-5b9fcc5fc8-tkqdr" event={"ID":"d17a46f4-1ca4-4674-8879-b82ff3e7df10","Type":"ContainerDied","Data":"e4246797ec76352c3c7e6f53397d1379b95ff6c54947eb59a43509dfddb1b3db"}
Jul 24 17:19:06.734041 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.734015    2563 generic.go:356] "Generic (PLEG): container finished" podID="13dcb0d8-b891-4f9c-aedd-4dfe1cdcc6a3" containerID="47469a65a7e8a545101772146b4cd0127e0f212198cba6126632a9ee61ca4476" exitCode=0
Jul 24 17:19:06.734159 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.734062    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e30ccfde-9f66f9bbd-bjsh4"
Jul 24 17:19:06.734159 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.734074    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e30ccfde-9f66f9bbd-bjsh4" event={"ID":"13dcb0d8-b891-4f9c-aedd-4dfe1cdcc6a3","Type":"ContainerDied","Data":"47469a65a7e8a545101772146b4cd0127e0f212198cba6126632a9ee61ca4476"}
Jul 24 17:19:06.734159 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.734110    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e30ccfde-9f66f9bbd-bjsh4" event={"ID":"13dcb0d8-b891-4f9c-aedd-4dfe1cdcc6a3","Type":"ContainerDied","Data":"343549290c028a908c7b76769d39809a329a49a06a4d4d97f98b9e83f076629c"}
Jul 24 17:19:06.742358 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.742341    2563 scope.go:117] "RemoveContainer" containerID="0039800a520748c40053dde20c93c98c9c8e36729af851416444002a04b3ef21"
Jul 24 17:19:06.750929 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.750912    2563 scope.go:117] "RemoveContainer" containerID="0039800a520748c40053dde20c93c98c9c8e36729af851416444002a04b3ef21"
Jul 24 17:19:06.751204 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:19:06.751186    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0039800a520748c40053dde20c93c98c9c8e36729af851416444002a04b3ef21\": container with ID starting with 0039800a520748c40053dde20c93c98c9c8e36729af851416444002a04b3ef21 not found: ID does not exist" containerID="0039800a520748c40053dde20c93c98c9c8e36729af851416444002a04b3ef21"
Jul 24 17:19:06.751248 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.751211    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0039800a520748c40053dde20c93c98c9c8e36729af851416444002a04b3ef21"} err="failed to get container status \"0039800a520748c40053dde20c93c98c9c8e36729af851416444002a04b3ef21\": rpc error: code = NotFound desc = could not find container \"0039800a520748c40053dde20c93c98c9c8e36729af851416444002a04b3ef21\": container with ID starting with 0039800a520748c40053dde20c93c98c9c8e36729af851416444002a04b3ef21 not found: ID does not exist"
Jul 24 17:19:06.751248 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.751226    2563 scope.go:117] "RemoveContainer" containerID="47469a65a7e8a545101772146b4cd0127e0f212198cba6126632a9ee61ca4476"
Jul 24 17:19:06.759339 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.759316    2563 scope.go:117] "RemoveContainer" containerID="47469a65a7e8a545101772146b4cd0127e0f212198cba6126632a9ee61ca4476"
Jul 24 17:19:06.759630 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:19:06.759609    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47469a65a7e8a545101772146b4cd0127e0f212198cba6126632a9ee61ca4476\": container with ID starting with 47469a65a7e8a545101772146b4cd0127e0f212198cba6126632a9ee61ca4476 not found: ID does not exist" containerID="47469a65a7e8a545101772146b4cd0127e0f212198cba6126632a9ee61ca4476"
Jul 24 17:19:06.759684 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.759641    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47469a65a7e8a545101772146b4cd0127e0f212198cba6126632a9ee61ca4476"} err="failed to get container status \"47469a65a7e8a545101772146b4cd0127e0f212198cba6126632a9ee61ca4476\": rpc error: code = NotFound desc = could not find container \"47469a65a7e8a545101772146b4cd0127e0f212198cba6126632a9ee61ca4476\": container with ID starting with 47469a65a7e8a545101772146b4cd0127e0f212198cba6126632a9ee61ca4476 not found: ID does not exist"
Jul 24 17:19:06.759865 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.759846    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e30ccfde-7c86988c55-8r8zh"]
Jul 24 17:19:06.762502 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.762481    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e30ccfde-7c86988c55-8r8zh"]
Jul 24 17:19:06.772196 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.772171    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e30ccfde-5b9fcc5fc8-tkqdr"]
Jul 24 17:19:06.775723 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.775704    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e30ccfde-5b9fcc5fc8-tkqdr"]
Jul 24 17:19:06.788470 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.788438    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e30ccfde-9f66f9bbd-bjsh4"]
Jul 24 17:19:06.793878 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:06.793855    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e30ccfde-9f66f9bbd-bjsh4"]
Jul 24 17:19:08.002789 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:08.002751    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13dcb0d8-b891-4f9c-aedd-4dfe1cdcc6a3" path="/var/lib/kubelet/pods/13dcb0d8-b891-4f9c-aedd-4dfe1cdcc6a3/volumes"
Jul 24 17:19:08.003208 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:08.003193    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc1b62d2-88b3-4679-99ad-546ad19bb589" path="/var/lib/kubelet/pods/bc1b62d2-88b3-4679-99ad-546ad19bb589/volumes"
Jul 24 17:19:08.003514 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:08.003502    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d17a46f4-1ca4-4674-8879-b82ff3e7df10" path="/var/lib/kubelet/pods/d17a46f4-1ca4-4674-8879-b82ff3e7df10/volumes"
Jul 24 17:19:10.112748 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:10.112711    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:19:10.495416 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:10.495383    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:19:12.012563 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:12.012522    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:19:16.314877 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:16.314844    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:19:16.368548 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:16.368523    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:19:16.614336 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:16.614258    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:19:16.824313 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:16.824278    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:19:16.959918 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:16.959888    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:19:17.063187 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:17.063152    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:19:17.248198 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:17.248107    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:19:22.798235 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:22.798200    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:19:22.907406 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:22.907375    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:19:22.998344 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:22.998309    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:19:23.161068 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:23.161033    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:19:28.689565 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:28.689506    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:19:28.753779 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:28.753745    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:19:35.603106 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:35.603066    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:19:35.936633 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:35.936598    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:19:36.499265 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:36.499236    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:19:38.617993 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:38.617955    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:19:38.703609 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:38.703574    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:19:38.708732 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:38.708709    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5e27a909-cc8b5fb64-sm4ng"]
Jul 24 17:19:38.709167 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:38.709146    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13dcb0d8-b891-4f9c-aedd-4dfe1cdcc6a3" containerName="payload-pre-processing"
Jul 24 17:19:38.709296 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:38.709242    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="13dcb0d8-b891-4f9c-aedd-4dfe1cdcc6a3" containerName="payload-pre-processing"
Jul 24 17:19:38.709296 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:38.709268    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d17a46f4-1ca4-4674-8879-b82ff3e7df10" containerName="payload-processing"
Jul 24 17:19:38.709296 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:38.709277    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d17a46f4-1ca4-4674-8879-b82ff3e7df10" containerName="payload-processing"
Jul 24 17:19:38.709469 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:38.709307    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc1b62d2-88b3-4679-99ad-546ad19bb589" containerName="maas-api"
Jul 24 17:19:38.709469 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:38.709315    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc1b62d2-88b3-4679-99ad-546ad19bb589" containerName="maas-api"
Jul 24 17:19:38.709469 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:38.709438    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="13dcb0d8-b891-4f9c-aedd-4dfe1cdcc6a3" containerName="payload-pre-processing"
Jul 24 17:19:38.709469 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:38.709456    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d17a46f4-1ca4-4674-8879-b82ff3e7df10" containerName="payload-processing"
Jul 24 17:19:38.709469 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:38.709468    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc1b62d2-88b3-4679-99ad-546ad19bb589" containerName="maas-api"
Jul 24 17:19:38.712591 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:38.712570    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e27a909-cc8b5fb64-sm4ng"
Jul 24 17:19:38.714714 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:38.714697    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-5e27a909-serving-cert\""
Jul 24 17:19:38.722867 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:38.722844    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5e27a909-cc8b5fb64-sm4ng"]
Jul 24 17:19:38.728554 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:38.728527    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f70b067-1b2f-4ecb-9999-455116a3febc-maas-api-tls\") pod \"maas-api-e2e-mt-5e27a909-cc8b5fb64-sm4ng\" (UID: \"3f70b067-1b2f-4ecb-9999-455116a3febc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e27a909-cc8b5fb64-sm4ng"
Jul 24 17:19:38.728665 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:38.728577    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f70b067-1b2f-4ecb-9999-455116a3febc-openshift-service-ca\") pod \"maas-api-e2e-mt-5e27a909-cc8b5fb64-sm4ng\" (UID: \"3f70b067-1b2f-4ecb-9999-455116a3febc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e27a909-cc8b5fb64-sm4ng"
Jul 24 17:19:38.728733 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:38.728696    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w45bp\" (UniqueName: \"kubernetes.io/projected/3f70b067-1b2f-4ecb-9999-455116a3febc-kube-api-access-w45bp\") pod \"maas-api-e2e-mt-5e27a909-cc8b5fb64-sm4ng\" (UID: \"3f70b067-1b2f-4ecb-9999-455116a3febc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e27a909-cc8b5fb64-sm4ng"
Jul 24 17:19:38.774381 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:38.774355    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:19:38.829113 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:38.829083    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w45bp\" (UniqueName: \"kubernetes.io/projected/3f70b067-1b2f-4ecb-9999-455116a3febc-kube-api-access-w45bp\") pod \"maas-api-e2e-mt-5e27a909-cc8b5fb64-sm4ng\" (UID: \"3f70b067-1b2f-4ecb-9999-455116a3febc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e27a909-cc8b5fb64-sm4ng"
Jul 24 17:19:38.829113 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:38.829115    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f70b067-1b2f-4ecb-9999-455116a3febc-maas-api-tls\") pod \"maas-api-e2e-mt-5e27a909-cc8b5fb64-sm4ng\" (UID: \"3f70b067-1b2f-4ecb-9999-455116a3febc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e27a909-cc8b5fb64-sm4ng"
Jul 24 17:19:38.829315 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:38.829153    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f70b067-1b2f-4ecb-9999-455116a3febc-openshift-service-ca\") pod \"maas-api-e2e-mt-5e27a909-cc8b5fb64-sm4ng\" (UID: \"3f70b067-1b2f-4ecb-9999-455116a3febc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e27a909-cc8b5fb64-sm4ng"
Jul 24 17:19:38.829315 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:19:38.829225    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-5e27a909-serving-cert: secret "maas-api-e2e-mt-5e27a909-serving-cert" not found
Jul 24 17:19:38.829315 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:19:38.829293    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f70b067-1b2f-4ecb-9999-455116a3febc-maas-api-tls podName:3f70b067-1b2f-4ecb-9999-455116a3febc nodeName:}" failed. No retries permitted until 2026-07-24 17:19:39.329278608 +0000 UTC m=+2201.971242470 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3f70b067-1b2f-4ecb-9999-455116a3febc-maas-api-tls") pod "maas-api-e2e-mt-5e27a909-cc8b5fb64-sm4ng" (UID: "3f70b067-1b2f-4ecb-9999-455116a3febc") : secret "maas-api-e2e-mt-5e27a909-serving-cert" not found
Jul 24 17:19:38.829765 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:38.829748    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f70b067-1b2f-4ecb-9999-455116a3febc-openshift-service-ca\") pod \"maas-api-e2e-mt-5e27a909-cc8b5fb64-sm4ng\" (UID: \"3f70b067-1b2f-4ecb-9999-455116a3febc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e27a909-cc8b5fb64-sm4ng"
Jul 24 17:19:38.837870 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:38.837845    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w45bp\" (UniqueName: \"kubernetes.io/projected/3f70b067-1b2f-4ecb-9999-455116a3febc-kube-api-access-w45bp\") pod \"maas-api-e2e-mt-5e27a909-cc8b5fb64-sm4ng\" (UID: \"3f70b067-1b2f-4ecb-9999-455116a3febc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e27a909-cc8b5fb64-sm4ng"
Jul 24 17:19:39.295393 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:39.295355    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:19:39.333822 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:39.333789    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f70b067-1b2f-4ecb-9999-455116a3febc-maas-api-tls\") pod \"maas-api-e2e-mt-5e27a909-cc8b5fb64-sm4ng\" (UID: \"3f70b067-1b2f-4ecb-9999-455116a3febc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e27a909-cc8b5fb64-sm4ng"
Jul 24 17:19:39.336463 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:39.336439    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f70b067-1b2f-4ecb-9999-455116a3febc-maas-api-tls\") pod \"maas-api-e2e-mt-5e27a909-cc8b5fb64-sm4ng\" (UID: \"3f70b067-1b2f-4ecb-9999-455116a3febc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e27a909-cc8b5fb64-sm4ng"
Jul 24 17:19:39.400258 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:39.400226    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:19:39.410253 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:39.410223    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5e27a909-69f6788776-mr2vp"]
Jul 24 17:19:39.414068 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:39.414051    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e27a909-69f6788776-mr2vp"
Jul 24 17:19:39.416462 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:39.416439    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-5e27a909-dockercfg-xczkk\""
Jul 24 17:19:39.416836 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:39.416819    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-5e27a909\""
Jul 24 17:19:39.424887 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:39.424865    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5e27a909-69f6788776-mr2vp"]
Jul 24 17:19:39.434357 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:39.434329    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1072999e-6927-43d9-be4a-42779d039124-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5e27a909-69f6788776-mr2vp\" (UID: \"1072999e-6927-43d9-be4a-42779d039124\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5e27a909-69f6788776-mr2vp"
Jul 24 17:19:39.434479 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:39.434368    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2thdh\" (UniqueName: \"kubernetes.io/projected/1072999e-6927-43d9-be4a-42779d039124-kube-api-access-2thdh\") pod \"payload-pre-processing-e2e-mt-5e27a909-69f6788776-mr2vp\" (UID: \"1072999e-6927-43d9-be4a-42779d039124\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5e27a909-69f6788776-mr2vp"
Jul 24 17:19:39.439099 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:39.439078    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5e27a909-59c87b9878-67ks8"]
Jul 24 17:19:39.442487 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:39.442468    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5e27a909-59c87b9878-67ks8"
Jul 24 17:19:39.452911 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:39.452879    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5e27a909-59c87b9878-67ks8"]
Jul 24 17:19:39.535420 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:39.535388    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1072999e-6927-43d9-be4a-42779d039124-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5e27a909-69f6788776-mr2vp\" (UID: \"1072999e-6927-43d9-be4a-42779d039124\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5e27a909-69f6788776-mr2vp"
Jul 24 17:19:39.535420 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:39.535423    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2thdh\" (UniqueName: \"kubernetes.io/projected/1072999e-6927-43d9-be4a-42779d039124-kube-api-access-2thdh\") pod \"payload-pre-processing-e2e-mt-5e27a909-69f6788776-mr2vp\" (UID: \"1072999e-6927-43d9-be4a-42779d039124\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5e27a909-69f6788776-mr2vp"
Jul 24 17:19:39.535619 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:39.535470    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1c5fadc-b33f-4225-a116-d9a671f82b6f-plugins-config-volume\") pod \"payload-processing-e2e-mt-5e27a909-59c87b9878-67ks8\" (UID: \"e1c5fadc-b33f-4225-a116-d9a671f82b6f\") " pod="openshift-ingress/payload-processing-e2e-mt-5e27a909-59c87b9878-67ks8"
Jul 24 17:19:39.535619 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:39.535513    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xsqb\" (UniqueName: \"kubernetes.io/projected/e1c5fadc-b33f-4225-a116-d9a671f82b6f-kube-api-access-5xsqb\") pod \"payload-processing-e2e-mt-5e27a909-59c87b9878-67ks8\" (UID: \"e1c5fadc-b33f-4225-a116-d9a671f82b6f\") " pod="openshift-ingress/payload-processing-e2e-mt-5e27a909-59c87b9878-67ks8"
Jul 24 17:19:39.536177 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:39.536157    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1072999e-6927-43d9-be4a-42779d039124-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5e27a909-69f6788776-mr2vp\" (UID: \"1072999e-6927-43d9-be4a-42779d039124\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5e27a909-69f6788776-mr2vp"
Jul 24 17:19:39.543038 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:39.543015    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2thdh\" (UniqueName: \"kubernetes.io/projected/1072999e-6927-43d9-be4a-42779d039124-kube-api-access-2thdh\") pod \"payload-pre-processing-e2e-mt-5e27a909-69f6788776-mr2vp\" (UID: \"1072999e-6927-43d9-be4a-42779d039124\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5e27a909-69f6788776-mr2vp"
Jul 24 17:19:39.624126 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:39.624034    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e27a909-cc8b5fb64-sm4ng"
Jul 24 17:19:39.636365 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:39.636336    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1c5fadc-b33f-4225-a116-d9a671f82b6f-plugins-config-volume\") pod \"payload-processing-e2e-mt-5e27a909-59c87b9878-67ks8\" (UID: \"e1c5fadc-b33f-4225-a116-d9a671f82b6f\") " pod="openshift-ingress/payload-processing-e2e-mt-5e27a909-59c87b9878-67ks8"
Jul 24 17:19:39.636482 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:39.636397    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xsqb\" (UniqueName: \"kubernetes.io/projected/e1c5fadc-b33f-4225-a116-d9a671f82b6f-kube-api-access-5xsqb\") pod \"payload-processing-e2e-mt-5e27a909-59c87b9878-67ks8\" (UID: \"e1c5fadc-b33f-4225-a116-d9a671f82b6f\") " pod="openshift-ingress/payload-processing-e2e-mt-5e27a909-59c87b9878-67ks8"
Jul 24 17:19:39.637143 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:39.637123    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1c5fadc-b33f-4225-a116-d9a671f82b6f-plugins-config-volume\") pod \"payload-processing-e2e-mt-5e27a909-59c87b9878-67ks8\" (UID: \"e1c5fadc-b33f-4225-a116-d9a671f82b6f\") " pod="openshift-ingress/payload-processing-e2e-mt-5e27a909-59c87b9878-67ks8"
Jul 24 17:19:39.644245 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:39.644222    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xsqb\" (UniqueName: \"kubernetes.io/projected/e1c5fadc-b33f-4225-a116-d9a671f82b6f-kube-api-access-5xsqb\") pod \"payload-processing-e2e-mt-5e27a909-59c87b9878-67ks8\" (UID: \"e1c5fadc-b33f-4225-a116-d9a671f82b6f\") " pod="openshift-ingress/payload-processing-e2e-mt-5e27a909-59c87b9878-67ks8"
Jul 24 17:19:39.725340 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:39.725244    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e27a909-69f6788776-mr2vp"
Jul 24 17:19:39.753754 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:39.753718    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5e27a909-59c87b9878-67ks8"
Jul 24 17:19:39.760654 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:39.760630    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5e27a909-cc8b5fb64-sm4ng"]
Jul 24 17:19:39.763864 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:19:39.763833    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f70b067_1b2f_4ecb_9999_455116a3febc.slice/crio-1f0f2776087798d157eec3045da7201b58d799a126b9765588c76853e56b8ac3 WatchSource:0}: Error finding container 1f0f2776087798d157eec3045da7201b58d799a126b9765588c76853e56b8ac3: Status 404 returned error can't find the container with id 1f0f2776087798d157eec3045da7201b58d799a126b9765588c76853e56b8ac3
Jul 24 17:19:39.765555 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:39.765539    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 17:19:39.902663 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:39.902623    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e27a909-cc8b5fb64-sm4ng" event={"ID":"3f70b067-1b2f-4ecb-9999-455116a3febc","Type":"ContainerStarted","Data":"1f0f2776087798d157eec3045da7201b58d799a126b9765588c76853e56b8ac3"}
Jul 24 17:19:39.903547 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:39.903512    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5e27a909-59c87b9878-67ks8"]
Jul 24 17:19:39.906581 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:19:39.906553    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1c5fadc_b33f_4225_a116_d9a671f82b6f.slice/crio-4feb0aa332bc0f8eb547cef64ca31187d5a212985c80ad63b7adf98bebe11b1c WatchSource:0}: Error finding container 4feb0aa332bc0f8eb547cef64ca31187d5a212985c80ad63b7adf98bebe11b1c: Status 404 returned error can't find the container with id 4feb0aa332bc0f8eb547cef64ca31187d5a212985c80ad63b7adf98bebe11b1c
Jul 24 17:19:40.078439 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:40.078398    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5e27a909-69f6788776-mr2vp"]
Jul 24 17:19:40.082104 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:19:40.082076    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1072999e_6927_43d9_be4a_42779d039124.slice/crio-2646102c44f9b8d90c8f4a8aa1aeb390508116d1554d2e07bd793835d222bd2a WatchSource:0}: Error finding container 2646102c44f9b8d90c8f4a8aa1aeb390508116d1554d2e07bd793835d222bd2a: Status 404 returned error can't find the container with id 2646102c44f9b8d90c8f4a8aa1aeb390508116d1554d2e07bd793835d222bd2a
Jul 24 17:19:40.909971 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:40.909918    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e27a909-cc8b5fb64-sm4ng" event={"ID":"3f70b067-1b2f-4ecb-9999-455116a3febc","Type":"ContainerStarted","Data":"434045b626ffdd45f1d33d947200c61f8afaa9367ba4d66a85e88a4354a4fe41"}
Jul 24 17:19:40.910415 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:40.910035    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e27a909-cc8b5fb64-sm4ng"
Jul 24 17:19:40.911470 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:40.911442    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5e27a909-59c87b9878-67ks8" event={"ID":"e1c5fadc-b33f-4225-a116-d9a671f82b6f","Type":"ContainerStarted","Data":"e1951318dac0e6915aeb17f2246b3d0ea5b5a5aedb8d4a3238d39b315feacaa0"}
Jul 24 17:19:40.911595 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:40.911491    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5e27a909-59c87b9878-67ks8" event={"ID":"e1c5fadc-b33f-4225-a116-d9a671f82b6f","Type":"ContainerStarted","Data":"4feb0aa332bc0f8eb547cef64ca31187d5a212985c80ad63b7adf98bebe11b1c"}
Jul 24 17:19:40.911595 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:40.911522    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-5e27a909-59c87b9878-67ks8"
Jul 24 17:19:40.913046 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:40.913024    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e27a909-69f6788776-mr2vp" event={"ID":"1072999e-6927-43d9-be4a-42779d039124","Type":"ContainerStarted","Data":"1e51f0b23dda523bd5222988ad3a40dabdaf1468928e4bc3f0f607623cf3900b"}
Jul 24 17:19:40.913152 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:40.913065    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e27a909-69f6788776-mr2vp" event={"ID":"1072999e-6927-43d9-be4a-42779d039124","Type":"ContainerStarted","Data":"2646102c44f9b8d90c8f4a8aa1aeb390508116d1554d2e07bd793835d222bd2a"}
Jul 24 17:19:40.913152 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:40.913128    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e27a909-69f6788776-mr2vp"
Jul 24 17:19:40.927351 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:40.927308    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e27a909-cc8b5fb64-sm4ng" podStartSLOduration=2.566163496 podStartE2EDuration="2.927294637s" totalImagesPullingTime="361.131141ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:19:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:19:40.925227637 +0000 UTC m=+2203.567191542" watchObservedRunningTime="2026-07-24 17:19:40.927294637 +0000 UTC m=+2203.569258521"
Jul 24 17:19:40.941917 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:40.941865    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-5e27a909-59c87b9878-67ks8" podStartSLOduration=1.941844658 podStartE2EDuration="1.941844658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:19:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:19:40.940666828 +0000 UTC m=+2203.582630739" watchObservedRunningTime="2026-07-24 17:19:40.941844658 +0000 UTC m=+2203.583808545"
Jul 24 17:19:40.968332 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:40.968285    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e27a909-69f6788776-mr2vp" podStartSLOduration=1.968270081 podStartE2EDuration="1.968270081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:19:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:19:40.966564359 +0000 UTC m=+2203.608528263" watchObservedRunningTime="2026-07-24 17:19:40.968270081 +0000 UTC m=+2203.610233965"
Jul 24 17:19:43.143952 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:43.143912    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:19:46.923196 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:46.923167    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e27a909-cc8b5fb64-sm4ng"
Jul 24 17:19:49.881397 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:49.881362    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:19:49.981898 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:49.981863    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:19:51.920505 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:51.920474    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e27a909-69f6788776-mr2vp"
Jul 24 17:19:51.923374 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:51.920554    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-5e27a909-59c87b9878-67ks8"
Jul 24 17:19:56.229173 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:19:56.229070    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:20:33.232884 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:33.232789    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5e27a909-cc8b5fb64-sm4ng"]
Jul 24 17:20:33.233857 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:33.233064    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e27a909-cc8b5fb64-sm4ng" podUID="3f70b067-1b2f-4ecb-9999-455116a3febc" containerName="maas-api" containerID="cri-o://434045b626ffdd45f1d33d947200c61f8afaa9367ba4d66a85e88a4354a4fe41" gracePeriod=30
Jul 24 17:20:33.255658 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:33.255622    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:20:33.283756 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:33.283723    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5e27a909-59c87b9878-67ks8"]
Jul 24 17:20:33.283989 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:33.283959    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-5e27a909-59c87b9878-67ks8" podUID="e1c5fadc-b33f-4225-a116-d9a671f82b6f" containerName="payload-processing" containerID="cri-o://e1951318dac0e6915aeb17f2246b3d0ea5b5a5aedb8d4a3238d39b315feacaa0" gracePeriod=30
Jul 24 17:20:33.291525 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:33.291491    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5e27a909-69f6788776-mr2vp"]
Jul 24 17:20:33.291720 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:33.291698    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e27a909-69f6788776-mr2vp" podUID="1072999e-6927-43d9-be4a-42779d039124" containerName="payload-pre-processing" containerID="cri-o://1e51f0b23dda523bd5222988ad3a40dabdaf1468928e4bc3f0f607623cf3900b" gracePeriod=30
Jul 24 17:20:33.325078 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:33.325036    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:20:33.424484 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:33.424438    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:20:33.578303 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:33.578268    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5e27a909-59c87b9878-67ks8"
Jul 24 17:20:33.581571 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:33.581549    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e27a909-cc8b5fb64-sm4ng"
Jul 24 17:20:33.589675 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:33.589654    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e27a909-69f6788776-mr2vp"
Jul 24 17:20:33.626621 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:33.626588    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f70b067-1b2f-4ecb-9999-455116a3febc-openshift-service-ca\") pod \"3f70b067-1b2f-4ecb-9999-455116a3febc\" (UID: \"3f70b067-1b2f-4ecb-9999-455116a3febc\") "
Jul 24 17:20:33.626989 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:33.626929    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2thdh\" (UniqueName: \"kubernetes.io/projected/1072999e-6927-43d9-be4a-42779d039124-kube-api-access-2thdh\") pod \"1072999e-6927-43d9-be4a-42779d039124\" (UID: \"1072999e-6927-43d9-be4a-42779d039124\") "
Jul 24 17:20:33.628129 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:33.627007    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f70b067-1b2f-4ecb-9999-455116a3febc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3f70b067-1b2f-4ecb-9999-455116a3febc" (UID: "3f70b067-1b2f-4ecb-9999-455116a3febc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:20:33.628129 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:33.627212    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f70b067-1b2f-4ecb-9999-455116a3febc-maas-api-tls\") pod \"3f70b067-1b2f-4ecb-9999-455116a3febc\" (UID: \"3f70b067-1b2f-4ecb-9999-455116a3febc\") "
Jul 24 17:20:33.628129 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:33.627280    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1c5fadc-b33f-4225-a116-d9a671f82b6f-plugins-config-volume\") pod \"e1c5fadc-b33f-4225-a116-d9a671f82b6f\" (UID: \"e1c5fadc-b33f-4225-a116-d9a671f82b6f\") "
Jul 24 17:20:33.628129 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:33.627319    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xsqb\" (UniqueName: \"kubernetes.io/projected/e1c5fadc-b33f-4225-a116-d9a671f82b6f-kube-api-access-5xsqb\") pod \"e1c5fadc-b33f-4225-a116-d9a671f82b6f\" (UID: \"e1c5fadc-b33f-4225-a116-d9a671f82b6f\") "
Jul 24 17:20:33.628129 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:33.627350    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w45bp\" (UniqueName: \"kubernetes.io/projected/3f70b067-1b2f-4ecb-9999-455116a3febc-kube-api-access-w45bp\") pod \"3f70b067-1b2f-4ecb-9999-455116a3febc\" (UID: \"3f70b067-1b2f-4ecb-9999-455116a3febc\") "
Jul 24 17:20:33.628129 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:33.627394    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1072999e-6927-43d9-be4a-42779d039124-plugins-config-volume\") pod \"1072999e-6927-43d9-be4a-42779d039124\" (UID: \"1072999e-6927-43d9-be4a-42779d039124\") "
Jul 24 17:20:33.628129 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:33.627674    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f70b067-1b2f-4ecb-9999-455116a3febc-openshift-service-ca\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:20:33.628129 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:33.627712    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1c5fadc-b33f-4225-a116-d9a671f82b6f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e1c5fadc-b33f-4225-a116-d9a671f82b6f" (UID: "e1c5fadc-b33f-4225-a116-d9a671f82b6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:20:33.628129 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:33.628068    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1072999e-6927-43d9-be4a-42779d039124-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1072999e-6927-43d9-be4a-42779d039124" (UID: "1072999e-6927-43d9-be4a-42779d039124"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:20:33.629888 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:33.629859    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f70b067-1b2f-4ecb-9999-455116a3febc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3f70b067-1b2f-4ecb-9999-455116a3febc" (UID: "3f70b067-1b2f-4ecb-9999-455116a3febc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:20:33.630156 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:33.630118    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1c5fadc-b33f-4225-a116-d9a671f82b6f-kube-api-access-5xsqb" (OuterVolumeSpecName: "kube-api-access-5xsqb") pod "e1c5fadc-b33f-4225-a116-d9a671f82b6f" (UID: "e1c5fadc-b33f-4225-a116-d9a671f82b6f"). InnerVolumeSpecName "kube-api-access-5xsqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:20:33.630598 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:33.630577    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f70b067-1b2f-4ecb-9999-455116a3febc-kube-api-access-w45bp" (OuterVolumeSpecName: "kube-api-access-w45bp") pod "3f70b067-1b2f-4ecb-9999-455116a3febc" (UID: "3f70b067-1b2f-4ecb-9999-455116a3febc"). InnerVolumeSpecName "kube-api-access-w45bp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:20:33.631428 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:33.631406    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1072999e-6927-43d9-be4a-42779d039124-kube-api-access-2thdh" (OuterVolumeSpecName: "kube-api-access-2thdh") pod "1072999e-6927-43d9-be4a-42779d039124" (UID: "1072999e-6927-43d9-be4a-42779d039124"). InnerVolumeSpecName "kube-api-access-2thdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:20:33.729103 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:33.729064    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w45bp\" (UniqueName: \"kubernetes.io/projected/3f70b067-1b2f-4ecb-9999-455116a3febc-kube-api-access-w45bp\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:20:33.729103 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:33.729099    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1072999e-6927-43d9-be4a-42779d039124-plugins-config-volume\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:20:33.729103 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:33.729109    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2thdh\" (UniqueName: \"kubernetes.io/projected/1072999e-6927-43d9-be4a-42779d039124-kube-api-access-2thdh\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:20:33.729359 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:33.729119    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f70b067-1b2f-4ecb-9999-455116a3febc-maas-api-tls\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:20:33.729359 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:33.729128    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1c5fadc-b33f-4225-a116-d9a671f82b6f-plugins-config-volume\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:20:33.729359 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:33.729136    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xsqb\" (UniqueName: \"kubernetes.io/projected/e1c5fadc-b33f-4225-a116-d9a671f82b6f-kube-api-access-5xsqb\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:20:34.039384 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:20:34.039339    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f70b067_1b2f_4ecb_9999_455116a3febc.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1c5fadc_b33f_4225_a116_d9a671f82b6f.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f70b067_1b2f_4ecb_9999_455116a3febc.slice/crio-1f0f2776087798d157eec3045da7201b58d799a126b9765588c76853e56b8ac3\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1072999e_6927_43d9_be4a_42779d039124.slice\": RecentStats: unable to find data in memory cache]"
Jul 24 17:20:34.200785 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:34.200749    2563 generic.go:356] "Generic (PLEG): container finished" podID="e1c5fadc-b33f-4225-a116-d9a671f82b6f" containerID="e1951318dac0e6915aeb17f2246b3d0ea5b5a5aedb8d4a3238d39b315feacaa0" exitCode=0
Jul 24 17:20:34.201043 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:34.200808    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5e27a909-59c87b9878-67ks8"
Jul 24 17:20:34.201043 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:34.200844    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5e27a909-59c87b9878-67ks8" event={"ID":"e1c5fadc-b33f-4225-a116-d9a671f82b6f","Type":"ContainerDied","Data":"e1951318dac0e6915aeb17f2246b3d0ea5b5a5aedb8d4a3238d39b315feacaa0"}
Jul 24 17:20:34.201043 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:34.200965    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5e27a909-59c87b9878-67ks8" event={"ID":"e1c5fadc-b33f-4225-a116-d9a671f82b6f","Type":"ContainerDied","Data":"4feb0aa332bc0f8eb547cef64ca31187d5a212985c80ad63b7adf98bebe11b1c"}
Jul 24 17:20:34.201043 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:34.201003    2563 scope.go:117] "RemoveContainer" containerID="e1951318dac0e6915aeb17f2246b3d0ea5b5a5aedb8d4a3238d39b315feacaa0"
Jul 24 17:20:34.202199 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:34.202095    2563 generic.go:356] "Generic (PLEG): container finished" podID="1072999e-6927-43d9-be4a-42779d039124" containerID="1e51f0b23dda523bd5222988ad3a40dabdaf1468928e4bc3f0f607623cf3900b" exitCode=0
Jul 24 17:20:34.202199 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:34.202159    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e27a909-69f6788776-mr2vp"
Jul 24 17:20:34.202199 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:34.202177    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e27a909-69f6788776-mr2vp" event={"ID":"1072999e-6927-43d9-be4a-42779d039124","Type":"ContainerDied","Data":"1e51f0b23dda523bd5222988ad3a40dabdaf1468928e4bc3f0f607623cf3900b"}
Jul 24 17:20:34.202394 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:34.202226    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e27a909-69f6788776-mr2vp" event={"ID":"1072999e-6927-43d9-be4a-42779d039124","Type":"ContainerDied","Data":"2646102c44f9b8d90c8f4a8aa1aeb390508116d1554d2e07bd793835d222bd2a"}
Jul 24 17:20:34.203856 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:34.203830    2563 generic.go:356] "Generic (PLEG): container finished" podID="3f70b067-1b2f-4ecb-9999-455116a3febc" containerID="434045b626ffdd45f1d33d947200c61f8afaa9367ba4d66a85e88a4354a4fe41" exitCode=0
Jul 24 17:20:34.203978 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:34.203958    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e27a909-cc8b5fb64-sm4ng"
Jul 24 17:20:34.203978 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:34.203968    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e27a909-cc8b5fb64-sm4ng" event={"ID":"3f70b067-1b2f-4ecb-9999-455116a3febc","Type":"ContainerDied","Data":"434045b626ffdd45f1d33d947200c61f8afaa9367ba4d66a85e88a4354a4fe41"}
Jul 24 17:20:34.204122 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:34.204011    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e27a909-cc8b5fb64-sm4ng" event={"ID":"3f70b067-1b2f-4ecb-9999-455116a3febc","Type":"ContainerDied","Data":"1f0f2776087798d157eec3045da7201b58d799a126b9765588c76853e56b8ac3"}
Jul 24 17:20:34.210771 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:34.210753    2563 scope.go:117] "RemoveContainer" containerID="e1951318dac0e6915aeb17f2246b3d0ea5b5a5aedb8d4a3238d39b315feacaa0"
Jul 24 17:20:34.211067 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:20:34.211045    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1951318dac0e6915aeb17f2246b3d0ea5b5a5aedb8d4a3238d39b315feacaa0\": container with ID starting with e1951318dac0e6915aeb17f2246b3d0ea5b5a5aedb8d4a3238d39b315feacaa0 not found: ID does not exist" containerID="e1951318dac0e6915aeb17f2246b3d0ea5b5a5aedb8d4a3238d39b315feacaa0"
Jul 24 17:20:34.211129 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:34.211076    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1951318dac0e6915aeb17f2246b3d0ea5b5a5aedb8d4a3238d39b315feacaa0"} err="failed to get container status \"e1951318dac0e6915aeb17f2246b3d0ea5b5a5aedb8d4a3238d39b315feacaa0\": rpc error: code = NotFound desc = could not find container \"e1951318dac0e6915aeb17f2246b3d0ea5b5a5aedb8d4a3238d39b315feacaa0\": container with ID starting with e1951318dac0e6915aeb17f2246b3d0ea5b5a5aedb8d4a3238d39b315feacaa0 not found: ID does not exist"
Jul 24 17:20:34.211129 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:34.211094    2563 scope.go:117] "RemoveContainer" containerID="1e51f0b23dda523bd5222988ad3a40dabdaf1468928e4bc3f0f607623cf3900b"
Jul 24 17:20:34.219390 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:34.219372    2563 scope.go:117] "RemoveContainer" containerID="1e51f0b23dda523bd5222988ad3a40dabdaf1468928e4bc3f0f607623cf3900b"
Jul 24 17:20:34.219670 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:20:34.219649    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e51f0b23dda523bd5222988ad3a40dabdaf1468928e4bc3f0f607623cf3900b\": container with ID starting with 1e51f0b23dda523bd5222988ad3a40dabdaf1468928e4bc3f0f607623cf3900b not found: ID does not exist" containerID="1e51f0b23dda523bd5222988ad3a40dabdaf1468928e4bc3f0f607623cf3900b"
Jul 24 17:20:34.219748 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:34.219675    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e51f0b23dda523bd5222988ad3a40dabdaf1468928e4bc3f0f607623cf3900b"} err="failed to get container status \"1e51f0b23dda523bd5222988ad3a40dabdaf1468928e4bc3f0f607623cf3900b\": rpc error: code = NotFound desc = could not find container \"1e51f0b23dda523bd5222988ad3a40dabdaf1468928e4bc3f0f607623cf3900b\": container with ID starting with 1e51f0b23dda523bd5222988ad3a40dabdaf1468928e4bc3f0f607623cf3900b not found: ID does not exist"
Jul 24 17:20:34.219748 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:34.219692    2563 scope.go:117] "RemoveContainer" containerID="434045b626ffdd45f1d33d947200c61f8afaa9367ba4d66a85e88a4354a4fe41"
Jul 24 17:20:34.224703 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:34.224677    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5e27a909-59c87b9878-67ks8"]
Jul 24 17:20:34.229039 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:34.229016    2563 scope.go:117] "RemoveContainer" containerID="434045b626ffdd45f1d33d947200c61f8afaa9367ba4d66a85e88a4354a4fe41"
Jul 24 17:20:34.229168 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:34.229079    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5e27a909-59c87b9878-67ks8"]
Jul 24 17:20:34.229367 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:20:34.229346    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"434045b626ffdd45f1d33d947200c61f8afaa9367ba4d66a85e88a4354a4fe41\": container with ID starting with 434045b626ffdd45f1d33d947200c61f8afaa9367ba4d66a85e88a4354a4fe41 not found: ID does not exist" containerID="434045b626ffdd45f1d33d947200c61f8afaa9367ba4d66a85e88a4354a4fe41"
Jul 24 17:20:34.229415 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:34.229374    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"434045b626ffdd45f1d33d947200c61f8afaa9367ba4d66a85e88a4354a4fe41"} err="failed to get container status \"434045b626ffdd45f1d33d947200c61f8afaa9367ba4d66a85e88a4354a4fe41\": rpc error: code = NotFound desc = could not find container \"434045b626ffdd45f1d33d947200c61f8afaa9367ba4d66a85e88a4354a4fe41\": container with ID starting with 434045b626ffdd45f1d33d947200c61f8afaa9367ba4d66a85e88a4354a4fe41 not found: ID does not exist"
Jul 24 17:20:34.239135 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:34.239107    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5e27a909-69f6788776-mr2vp"]
Jul 24 17:20:34.244266 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:34.244237    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5e27a909-69f6788776-mr2vp"]
Jul 24 17:20:34.253884 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:34.253857    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5e27a909-cc8b5fb64-sm4ng"]
Jul 24 17:20:34.256921 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:34.256896    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5e27a909-cc8b5fb64-sm4ng"]
Jul 24 17:20:36.002851 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:36.002807    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1072999e-6927-43d9-be4a-42779d039124" path="/var/lib/kubelet/pods/1072999e-6927-43d9-be4a-42779d039124/volumes"
Jul 24 17:20:36.003389 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:36.003370    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f70b067-1b2f-4ecb-9999-455116a3febc" path="/var/lib/kubelet/pods/3f70b067-1b2f-4ecb-9999-455116a3febc/volumes"
Jul 24 17:20:36.003770 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:36.003751    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1c5fadc-b33f-4225-a116-d9a671f82b6f" path="/var/lib/kubelet/pods/e1c5fadc-b33f-4225-a116-d9a671f82b6f/volumes"
Jul 24 17:20:37.767588 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:37.767540    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:20:41.119085 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:41.119049    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:20:46.149345 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:20:46.149313    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:21:22.757247 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:21:22.757213    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:21:23.114865 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:21:23.114779    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:21:28.669784 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:21:28.669744    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:21:34.437078 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:21:34.437040    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:21:34.532677 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:21:34.532643    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:21:34.721175 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:21:34.721082    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:21:34.947044 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:21:34.947004    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:21:36.075875 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:21:36.075842    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:21:36.166515 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:21:36.166481    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:21:36.267054 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:21:36.267023    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:21:37.020018 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:21:37.019985    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:21:37.117601 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:21:37.117560    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:21:37.217695 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:21:37.217656    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:21:38.574120 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:21:38.574087    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:21:44.577287 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:21:44.577248    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:21:44.643378 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:21:44.643348    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:21:45.255155 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:21:45.255119    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:21:51.763903 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:21:51.763813    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:21:51.873450 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:21:51.873393    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:21:51.980821 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:21:51.980786    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:22:02.498313 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:02.498265    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:22:02.858528 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:02.858444    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:22:23.805586 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:23.805549    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:22:30.360994 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:30.360955    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:22:30.461220 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:30.461190    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:22:30.751664 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:30.751632    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:22:31.026371 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:31.026298    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:22:31.412014 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:31.411980    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:22:33.628831 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:33.628802    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:22:33.722923 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:33.722890    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:22:33.805362 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:33.805332    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0a7def-674b49955-546z6"]
Jul 24 17:22:33.805692 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:33.805681    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1c5fadc-b33f-4225-a116-d9a671f82b6f" containerName="payload-processing"
Jul 24 17:22:33.805747 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:33.805693    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1c5fadc-b33f-4225-a116-d9a671f82b6f" containerName="payload-processing"
Jul 24 17:22:33.805747 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:33.805707    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f70b067-1b2f-4ecb-9999-455116a3febc" containerName="maas-api"
Jul 24 17:22:33.805747 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:33.805713    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f70b067-1b2f-4ecb-9999-455116a3febc" containerName="maas-api"
Jul 24 17:22:33.805747 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:33.805735    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1072999e-6927-43d9-be4a-42779d039124" containerName="payload-pre-processing"
Jul 24 17:22:33.805747 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:33.805741    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1072999e-6927-43d9-be4a-42779d039124" containerName="payload-pre-processing"
Jul 24 17:22:33.805922 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:33.805805    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f70b067-1b2f-4ecb-9999-455116a3febc" containerName="maas-api"
Jul 24 17:22:33.805922 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:33.805815    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1072999e-6927-43d9-be4a-42779d039124" containerName="payload-pre-processing"
Jul 24 17:22:33.805922 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:33.805822    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1c5fadc-b33f-4225-a116-d9a671f82b6f" containerName="payload-processing"
Jul 24 17:22:33.808978 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:33.808956    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a7def-674b49955-546z6"
Jul 24 17:22:33.811264 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:33.811238    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-0a7def-serving-cert\""
Jul 24 17:22:33.822492 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:33.822468    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0a7def-674b49955-546z6"]
Jul 24 17:22:33.842286 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:33.842263    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:22:33.868144 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:33.868116    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr4d5\" (UniqueName: \"kubernetes.io/projected/634d9ea5-ec24-47a3-bdc8-984d1517af22-kube-api-access-fr4d5\") pod \"maas-api-e2e-inf-a-0a7def-674b49955-546z6\" (UID: \"634d9ea5-ec24-47a3-bdc8-984d1517af22\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a7def-674b49955-546z6"
Jul 24 17:22:33.868263 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:33.868166    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/634d9ea5-ec24-47a3-bdc8-984d1517af22-maas-api-tls\") pod \"maas-api-e2e-inf-a-0a7def-674b49955-546z6\" (UID: \"634d9ea5-ec24-47a3-bdc8-984d1517af22\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a7def-674b49955-546z6"
Jul 24 17:22:33.868263 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:33.868212    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/634d9ea5-ec24-47a3-bdc8-984d1517af22-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0a7def-674b49955-546z6\" (UID: \"634d9ea5-ec24-47a3-bdc8-984d1517af22\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a7def-674b49955-546z6"
Jul 24 17:22:33.968787 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:33.968754    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr4d5\" (UniqueName: \"kubernetes.io/projected/634d9ea5-ec24-47a3-bdc8-984d1517af22-kube-api-access-fr4d5\") pod \"maas-api-e2e-inf-a-0a7def-674b49955-546z6\" (UID: \"634d9ea5-ec24-47a3-bdc8-984d1517af22\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a7def-674b49955-546z6"
Jul 24 17:22:33.968913 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:33.968802    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/634d9ea5-ec24-47a3-bdc8-984d1517af22-maas-api-tls\") pod \"maas-api-e2e-inf-a-0a7def-674b49955-546z6\" (UID: \"634d9ea5-ec24-47a3-bdc8-984d1517af22\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a7def-674b49955-546z6"
Jul 24 17:22:33.968913 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:22:33.968894    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-0a7def-serving-cert: secret "maas-api-e2e-inf-a-0a7def-serving-cert" not found
Jul 24 17:22:33.969046 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:33.968915    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/634d9ea5-ec24-47a3-bdc8-984d1517af22-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0a7def-674b49955-546z6\" (UID: \"634d9ea5-ec24-47a3-bdc8-984d1517af22\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a7def-674b49955-546z6"
Jul 24 17:22:33.969046 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:22:33.968987    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/634d9ea5-ec24-47a3-bdc8-984d1517af22-maas-api-tls podName:634d9ea5-ec24-47a3-bdc8-984d1517af22 nodeName:}" failed. No retries permitted until 2026-07-24 17:22:34.46896662 +0000 UTC m=+2377.110930499 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/634d9ea5-ec24-47a3-bdc8-984d1517af22-maas-api-tls") pod "maas-api-e2e-inf-a-0a7def-674b49955-546z6" (UID: "634d9ea5-ec24-47a3-bdc8-984d1517af22") : secret "maas-api-e2e-inf-a-0a7def-serving-cert" not found
Jul 24 17:22:33.969501 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:33.969482    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/634d9ea5-ec24-47a3-bdc8-984d1517af22-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0a7def-674b49955-546z6\" (UID: \"634d9ea5-ec24-47a3-bdc8-984d1517af22\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a7def-674b49955-546z6"
Jul 24 17:22:33.977835 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:33.977816    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr4d5\" (UniqueName: \"kubernetes.io/projected/634d9ea5-ec24-47a3-bdc8-984d1517af22-kube-api-access-fr4d5\") pod \"maas-api-e2e-inf-a-0a7def-674b49955-546z6\" (UID: \"634d9ea5-ec24-47a3-bdc8-984d1517af22\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a7def-674b49955-546z6"
Jul 24 17:22:34.423778 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:34.423741    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:22:34.471997 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:34.471965    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/634d9ea5-ec24-47a3-bdc8-984d1517af22-maas-api-tls\") pod \"maas-api-e2e-inf-a-0a7def-674b49955-546z6\" (UID: \"634d9ea5-ec24-47a3-bdc8-984d1517af22\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a7def-674b49955-546z6"
Jul 24 17:22:34.472156 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:22:34.472116    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-0a7def-serving-cert: secret "maas-api-e2e-inf-a-0a7def-serving-cert" not found
Jul 24 17:22:34.472201 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:22:34.472175    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/634d9ea5-ec24-47a3-bdc8-984d1517af22-maas-api-tls podName:634d9ea5-ec24-47a3-bdc8-984d1517af22 nodeName:}" failed. No retries permitted until 2026-07-24 17:22:35.472160438 +0000 UTC m=+2378.114124304 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/634d9ea5-ec24-47a3-bdc8-984d1517af22-maas-api-tls") pod "maas-api-e2e-inf-a-0a7def-674b49955-546z6" (UID: "634d9ea5-ec24-47a3-bdc8-984d1517af22") : secret "maas-api-e2e-inf-a-0a7def-serving-cert" not found
Jul 24 17:22:34.523147 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:34.523116    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:22:34.549069 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:34.549043    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0a7def-66bbdb8ff6-cws6n"]
Jul 24 17:22:34.552371 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:34.552356    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0a7def-66bbdb8ff6-cws6n"
Jul 24 17:22:34.555301 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:34.555272    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-0a7def-dockercfg-ftjwm\""
Jul 24 17:22:34.555409 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:34.555334    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-0a7def\""
Jul 24 17:22:34.575071 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:34.575047    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0a7def-66bbdb8ff6-cws6n"]
Jul 24 17:22:34.604954 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:34.600340    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0a7def-778dcd6668-r7pf9"]
Jul 24 17:22:34.610242 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:34.610213    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0a7def-778dcd6668-r7pf9"
Jul 24 17:22:34.614778 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:34.614758    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0a7def-778dcd6668-r7pf9"]
Jul 24 17:22:34.674243 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:34.674160    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x69lp\" (UniqueName: \"kubernetes.io/projected/99d80ffc-aabf-4256-848d-02ac2be61fd9-kube-api-access-x69lp\") pod \"payload-pre-processing-e2e-inf-a-0a7def-778dcd6668-r7pf9\" (UID: \"99d80ffc-aabf-4256-848d-02ac2be61fd9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0a7def-778dcd6668-r7pf9"
Jul 24 17:22:34.674602 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:34.674277    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrkqs\" (UniqueName: \"kubernetes.io/projected/af12882e-1183-42ff-834f-7643d8f0c2ee-kube-api-access-hrkqs\") pod \"payload-processing-e2e-inf-a-0a7def-66bbdb8ff6-cws6n\" (UID: \"af12882e-1183-42ff-834f-7643d8f0c2ee\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0a7def-66bbdb8ff6-cws6n"
Jul 24 17:22:34.674602 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:34.674341    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af12882e-1183-42ff-834f-7643d8f0c2ee-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0a7def-66bbdb8ff6-cws6n\" (UID: \"af12882e-1183-42ff-834f-7643d8f0c2ee\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0a7def-66bbdb8ff6-cws6n"
Jul 24 17:22:34.674602 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:34.674367    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99d80ffc-aabf-4256-848d-02ac2be61fd9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0a7def-778dcd6668-r7pf9\" (UID: \"99d80ffc-aabf-4256-848d-02ac2be61fd9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0a7def-778dcd6668-r7pf9"
Jul 24 17:22:34.679111 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:34.679075    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:22:34.775641 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:34.775612    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrkqs\" (UniqueName: \"kubernetes.io/projected/af12882e-1183-42ff-834f-7643d8f0c2ee-kube-api-access-hrkqs\") pod \"payload-processing-e2e-inf-a-0a7def-66bbdb8ff6-cws6n\" (UID: \"af12882e-1183-42ff-834f-7643d8f0c2ee\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0a7def-66bbdb8ff6-cws6n"
Jul 24 17:22:34.775827 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:34.775664    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af12882e-1183-42ff-834f-7643d8f0c2ee-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0a7def-66bbdb8ff6-cws6n\" (UID: \"af12882e-1183-42ff-834f-7643d8f0c2ee\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0a7def-66bbdb8ff6-cws6n"
Jul 24 17:22:34.775827 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:34.775687    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99d80ffc-aabf-4256-848d-02ac2be61fd9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0a7def-778dcd6668-r7pf9\" (UID: \"99d80ffc-aabf-4256-848d-02ac2be61fd9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0a7def-778dcd6668-r7pf9"
Jul 24 17:22:34.775827 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:34.775749    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x69lp\" (UniqueName: \"kubernetes.io/projected/99d80ffc-aabf-4256-848d-02ac2be61fd9-kube-api-access-x69lp\") pod \"payload-pre-processing-e2e-inf-a-0a7def-778dcd6668-r7pf9\" (UID: \"99d80ffc-aabf-4256-848d-02ac2be61fd9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0a7def-778dcd6668-r7pf9"
Jul 24 17:22:34.776445 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:34.776422    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99d80ffc-aabf-4256-848d-02ac2be61fd9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0a7def-778dcd6668-r7pf9\" (UID: \"99d80ffc-aabf-4256-848d-02ac2be61fd9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0a7def-778dcd6668-r7pf9"
Jul 24 17:22:34.776578 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:34.776500    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af12882e-1183-42ff-834f-7643d8f0c2ee-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0a7def-66bbdb8ff6-cws6n\" (UID: \"af12882e-1183-42ff-834f-7643d8f0c2ee\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0a7def-66bbdb8ff6-cws6n"
Jul 24 17:22:34.783656 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:34.783627    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x69lp\" (UniqueName: \"kubernetes.io/projected/99d80ffc-aabf-4256-848d-02ac2be61fd9-kube-api-access-x69lp\") pod \"payload-pre-processing-e2e-inf-a-0a7def-778dcd6668-r7pf9\" (UID: \"99d80ffc-aabf-4256-848d-02ac2be61fd9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0a7def-778dcd6668-r7pf9"
Jul 24 17:22:34.784055 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:34.784036    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrkqs\" (UniqueName: \"kubernetes.io/projected/af12882e-1183-42ff-834f-7643d8f0c2ee-kube-api-access-hrkqs\") pod \"payload-processing-e2e-inf-a-0a7def-66bbdb8ff6-cws6n\" (UID: \"af12882e-1183-42ff-834f-7643d8f0c2ee\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0a7def-66bbdb8ff6-cws6n"
Jul 24 17:22:34.863364 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:34.863332    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0a7def-66bbdb8ff6-cws6n"
Jul 24 17:22:34.920902 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:34.920871    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0a7def-778dcd6668-r7pf9"
Jul 24 17:22:35.010603 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:35.010577    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0a7def-66bbdb8ff6-cws6n"]
Jul 24 17:22:35.013995 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:22:35.013963    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf12882e_1183_42ff_834f_7643d8f0c2ee.slice/crio-ff63f4518416259db56b6baebd812f81dcf7f527af650f12867d38040bc5cb90 WatchSource:0}: Error finding container ff63f4518416259db56b6baebd812f81dcf7f527af650f12867d38040bc5cb90: Status 404 returned error can't find the container with id ff63f4518416259db56b6baebd812f81dcf7f527af650f12867d38040bc5cb90
Jul 24 17:22:35.279242 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:35.279215    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0a7def-778dcd6668-r7pf9"]
Jul 24 17:22:35.281859 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:22:35.281826    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99d80ffc_aabf_4256_848d_02ac2be61fd9.slice/crio-06cf444e3050f15e7f2d7c556e942587388326ba5de771d2b55a51ab1a1d6a35 WatchSource:0}: Error finding container 06cf444e3050f15e7f2d7c556e942587388326ba5de771d2b55a51ab1a1d6a35: Status 404 returned error can't find the container with id 06cf444e3050f15e7f2d7c556e942587388326ba5de771d2b55a51ab1a1d6a35
Jul 24 17:22:35.483339 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:35.483296    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/634d9ea5-ec24-47a3-bdc8-984d1517af22-maas-api-tls\") pod \"maas-api-e2e-inf-a-0a7def-674b49955-546z6\" (UID: \"634d9ea5-ec24-47a3-bdc8-984d1517af22\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a7def-674b49955-546z6"
Jul 24 17:22:35.485801 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:35.485779    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/634d9ea5-ec24-47a3-bdc8-984d1517af22-maas-api-tls\") pod \"maas-api-e2e-inf-a-0a7def-674b49955-546z6\" (UID: \"634d9ea5-ec24-47a3-bdc8-984d1517af22\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a7def-674b49955-546z6"
Jul 24 17:22:35.618847 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:35.618760    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a7def-674b49955-546z6"
Jul 24 17:22:35.758350 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:35.758304    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0a7def-674b49955-546z6"]
Jul 24 17:22:35.760759 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:22:35.760731    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod634d9ea5_ec24_47a3_bdc8_984d1517af22.slice/crio-3974434b15a2304ee1b3e396e83f672b412b687535c84901cc5a3135ec93f7a5 WatchSource:0}: Error finding container 3974434b15a2304ee1b3e396e83f672b412b687535c84901cc5a3135ec93f7a5: Status 404 returned error can't find the container with id 3974434b15a2304ee1b3e396e83f672b412b687535c84901cc5a3135ec93f7a5
Jul 24 17:22:35.785621 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:35.785595    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0a7def-778dcd6668-r7pf9" event={"ID":"99d80ffc-aabf-4256-848d-02ac2be61fd9","Type":"ContainerStarted","Data":"ab2ae2e787d4d19a0a6a4a1a34dbb9deed29a8ed6736b1280adb6d336b2a6c43"}
Jul 24 17:22:35.785748 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:35.785644    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0a7def-778dcd6668-r7pf9" event={"ID":"99d80ffc-aabf-4256-848d-02ac2be61fd9","Type":"ContainerStarted","Data":"06cf444e3050f15e7f2d7c556e942587388326ba5de771d2b55a51ab1a1d6a35"}
Jul 24 17:22:35.785748 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:35.785695    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0a7def-778dcd6668-r7pf9"
Jul 24 17:22:35.786877 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:35.786851    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a7def-674b49955-546z6" event={"ID":"634d9ea5-ec24-47a3-bdc8-984d1517af22","Type":"ContainerStarted","Data":"3974434b15a2304ee1b3e396e83f672b412b687535c84901cc5a3135ec93f7a5"}
Jul 24 17:22:35.788382 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:35.788362    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0a7def-66bbdb8ff6-cws6n" event={"ID":"af12882e-1183-42ff-834f-7643d8f0c2ee","Type":"ContainerStarted","Data":"96bdd92f478929eed070966d3019f644943f0592f53c87a06390f2bebeae60cd"}
Jul 24 17:22:35.788467 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:35.788418    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0a7def-66bbdb8ff6-cws6n"
Jul 24 17:22:35.788467 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:35.788440    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0a7def-66bbdb8ff6-cws6n" event={"ID":"af12882e-1183-42ff-834f-7643d8f0c2ee","Type":"ContainerStarted","Data":"ff63f4518416259db56b6baebd812f81dcf7f527af650f12867d38040bc5cb90"}
Jul 24 17:22:35.803124 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:35.803084    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0a7def-778dcd6668-r7pf9" podStartSLOduration=1.803070862 podStartE2EDuration="1.803070862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:22:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:22:35.802219115 +0000 UTC m=+2378.444183054" watchObservedRunningTime="2026-07-24 17:22:35.803070862 +0000 UTC m=+2378.445034742"
Jul 24 17:22:35.819486 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:35.819444    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-0a7def-66bbdb8ff6-cws6n" podStartSLOduration=1.8194307090000001 podStartE2EDuration="1.819430709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:22:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:22:35.817520373 +0000 UTC m=+2378.459484272" watchObservedRunningTime="2026-07-24 17:22:35.819430709 +0000 UTC m=+2378.461394635"
Jul 24 17:22:36.795304 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:36.795210    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a7def-674b49955-546z6" event={"ID":"634d9ea5-ec24-47a3-bdc8-984d1517af22","Type":"ContainerStarted","Data":"d07e23e5162e589986662df8afc73965ec29c23c7a0775bb5da3e5de496d4285"}
Jul 24 17:22:36.795693 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:36.795444    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a7def-674b49955-546z6"
Jul 24 17:22:36.811683 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:36.811627    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a7def-674b49955-546z6" podStartSLOduration=3.149113207 podStartE2EDuration="3.811609735s" totalImagesPullingTime="662.496528ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:22:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:22:36.810809535 +0000 UTC m=+2379.452773432" watchObservedRunningTime="2026-07-24 17:22:36.811609735 +0000 UTC m=+2379.453573621"
Jul 24 17:22:38.136132 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:38.136094    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:22:38.525597 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:38.525508    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:22:38.787134 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:38.787054    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:22:39.091030 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:39.090920    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:22:41.733283 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:41.733244    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:22:41.825882 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:41.825841    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b3a7b4-7654ffb88b-rrxh4"]
Jul 24 17:22:41.828534 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:41.828507    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:22:41.830063 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:41.830042    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3a7b4-7654ffb88b-rrxh4"
Jul 24 17:22:41.833473 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:41.833450    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-b3a7b4-serving-cert\""
Jul 24 17:22:41.841311 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:41.841282    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b3a7b4-7654ffb88b-rrxh4"]
Jul 24 17:22:41.902135 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:41.902100    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:22:41.942125 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:41.942088    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c05f102-c4b1-400e-b664-6f587afe8cb8-maas-api-tls\") pod \"maas-api-e2e-inf-b-b3a7b4-7654ffb88b-rrxh4\" (UID: \"2c05f102-c4b1-400e-b664-6f587afe8cb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3a7b4-7654ffb88b-rrxh4"
Jul 24 17:22:41.942125 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:41.942132    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqxfn\" (UniqueName: \"kubernetes.io/projected/2c05f102-c4b1-400e-b664-6f587afe8cb8-kube-api-access-mqxfn\") pod \"maas-api-e2e-inf-b-b3a7b4-7654ffb88b-rrxh4\" (UID: \"2c05f102-c4b1-400e-b664-6f587afe8cb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3a7b4-7654ffb88b-rrxh4"
Jul 24 17:22:41.942334 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:41.942168    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c05f102-c4b1-400e-b664-6f587afe8cb8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b3a7b4-7654ffb88b-rrxh4\" (UID: \"2c05f102-c4b1-400e-b664-6f587afe8cb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3a7b4-7654ffb88b-rrxh4"
Jul 24 17:22:42.043240 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:42.043153    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c05f102-c4b1-400e-b664-6f587afe8cb8-maas-api-tls\") pod \"maas-api-e2e-inf-b-b3a7b4-7654ffb88b-rrxh4\" (UID: \"2c05f102-c4b1-400e-b664-6f587afe8cb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3a7b4-7654ffb88b-rrxh4"
Jul 24 17:22:42.043240 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:42.043191    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqxfn\" (UniqueName: \"kubernetes.io/projected/2c05f102-c4b1-400e-b664-6f587afe8cb8-kube-api-access-mqxfn\") pod \"maas-api-e2e-inf-b-b3a7b4-7654ffb88b-rrxh4\" (UID: \"2c05f102-c4b1-400e-b664-6f587afe8cb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3a7b4-7654ffb88b-rrxh4"
Jul 24 17:22:42.043503 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:42.043291    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c05f102-c4b1-400e-b664-6f587afe8cb8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b3a7b4-7654ffb88b-rrxh4\" (UID: \"2c05f102-c4b1-400e-b664-6f587afe8cb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3a7b4-7654ffb88b-rrxh4"
Jul 24 17:22:42.044129 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:42.044102    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c05f102-c4b1-400e-b664-6f587afe8cb8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b3a7b4-7654ffb88b-rrxh4\" (UID: \"2c05f102-c4b1-400e-b664-6f587afe8cb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3a7b4-7654ffb88b-rrxh4"
Jul 24 17:22:42.045961 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:42.045913    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c05f102-c4b1-400e-b664-6f587afe8cb8-maas-api-tls\") pod \"maas-api-e2e-inf-b-b3a7b4-7654ffb88b-rrxh4\" (UID: \"2c05f102-c4b1-400e-b664-6f587afe8cb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3a7b4-7654ffb88b-rrxh4"
Jul 24 17:22:42.059990 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:42.059970    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqxfn\" (UniqueName: \"kubernetes.io/projected/2c05f102-c4b1-400e-b664-6f587afe8cb8-kube-api-access-mqxfn\") pod \"maas-api-e2e-inf-b-b3a7b4-7654ffb88b-rrxh4\" (UID: \"2c05f102-c4b1-400e-b664-6f587afe8cb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3a7b4-7654ffb88b-rrxh4"
Jul 24 17:22:42.065376 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:42.065355    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:22:42.143895 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:42.143860    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3a7b4-7654ffb88b-rrxh4"
Jul 24 17:22:42.288992 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:42.288960    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b3a7b4-7654ffb88b-rrxh4"]
Jul 24 17:22:42.294146 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:22:42.294069    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c05f102_c4b1_400e_b664_6f587afe8cb8.slice/crio-4d9f93f5c0302471d9567a4012bcf68d01c8cd3cd5dd1fce76c2b94f9d114782 WatchSource:0}: Error finding container 4d9f93f5c0302471d9567a4012bcf68d01c8cd3cd5dd1fce76c2b94f9d114782: Status 404 returned error can't find the container with id 4d9f93f5c0302471d9567a4012bcf68d01c8cd3cd5dd1fce76c2b94f9d114782
Jul 24 17:22:42.425908 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:42.425865    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:22:42.524358 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:42.524314    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:22:42.533370 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:42.533340    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b3a7b4-d8869c4fb-wqgzn"]
Jul 24 17:22:42.538376 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:42.538354    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3a7b4-d8869c4fb-wqgzn"
Jul 24 17:22:42.540824 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:42.540801    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b3a7b4\""
Jul 24 17:22:42.541124 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:42.541106    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b3a7b4-dockercfg-wpx5m\""
Jul 24 17:22:42.553736 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:42.553710    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b3a7b4-d8869c4fb-wqgzn"]
Jul 24 17:22:42.565446 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:42.565413    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b3a7b4-5d64fcbfb6-6v88l"]
Jul 24 17:22:42.574400 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:42.574367    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b3a7b4-5d64fcbfb6-6v88l"
Jul 24 17:22:42.583735 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:42.583709    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b3a7b4-5d64fcbfb6-6v88l"]
Jul 24 17:22:42.651343 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:42.649623    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1308141-b7bf-4023-8a73-3af44251c97e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b3a7b4-5d64fcbfb6-6v88l\" (UID: \"a1308141-b7bf-4023-8a73-3af44251c97e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b3a7b4-5d64fcbfb6-6v88l"
Jul 24 17:22:42.651343 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:42.649704    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftbfn\" (UniqueName: \"kubernetes.io/projected/ecbc49dd-c78d-429c-bdbb-da551313a61a-kube-api-access-ftbfn\") pod \"payload-pre-processing-e2e-inf-b-b3a7b4-d8869c4fb-wqgzn\" (UID: \"ecbc49dd-c78d-429c-bdbb-da551313a61a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3a7b4-d8869c4fb-wqgzn"
Jul 24 17:22:42.651343 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:42.649780    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pnlm\" (UniqueName: \"kubernetes.io/projected/a1308141-b7bf-4023-8a73-3af44251c97e-kube-api-access-2pnlm\") pod \"payload-processing-e2e-inf-b-b3a7b4-5d64fcbfb6-6v88l\" (UID: \"a1308141-b7bf-4023-8a73-3af44251c97e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b3a7b4-5d64fcbfb6-6v88l"
Jul 24 17:22:42.651343 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:42.649996    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecbc49dd-c78d-429c-bdbb-da551313a61a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b3a7b4-d8869c4fb-wqgzn\" (UID: \"ecbc49dd-c78d-429c-bdbb-da551313a61a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3a7b4-d8869c4fb-wqgzn"
Jul 24 17:22:42.751455 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:42.751416    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecbc49dd-c78d-429c-bdbb-da551313a61a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b3a7b4-d8869c4fb-wqgzn\" (UID: \"ecbc49dd-c78d-429c-bdbb-da551313a61a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3a7b4-d8869c4fb-wqgzn"
Jul 24 17:22:42.751861 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:42.751520    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1308141-b7bf-4023-8a73-3af44251c97e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b3a7b4-5d64fcbfb6-6v88l\" (UID: \"a1308141-b7bf-4023-8a73-3af44251c97e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b3a7b4-5d64fcbfb6-6v88l"
Jul 24 17:22:42.751861 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:42.751568    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftbfn\" (UniqueName: \"kubernetes.io/projected/ecbc49dd-c78d-429c-bdbb-da551313a61a-kube-api-access-ftbfn\") pod \"payload-pre-processing-e2e-inf-b-b3a7b4-d8869c4fb-wqgzn\" (UID: \"ecbc49dd-c78d-429c-bdbb-da551313a61a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3a7b4-d8869c4fb-wqgzn"
Jul 24 17:22:42.751861 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:42.751619    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pnlm\" (UniqueName: \"kubernetes.io/projected/a1308141-b7bf-4023-8a73-3af44251c97e-kube-api-access-2pnlm\") pod \"payload-processing-e2e-inf-b-b3a7b4-5d64fcbfb6-6v88l\" (UID: \"a1308141-b7bf-4023-8a73-3af44251c97e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b3a7b4-5d64fcbfb6-6v88l"
Jul 24 17:22:42.752408 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:42.752385    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1308141-b7bf-4023-8a73-3af44251c97e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b3a7b4-5d64fcbfb6-6v88l\" (UID: \"a1308141-b7bf-4023-8a73-3af44251c97e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b3a7b4-5d64fcbfb6-6v88l"
Jul 24 17:22:42.752563 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:42.752533    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecbc49dd-c78d-429c-bdbb-da551313a61a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b3a7b4-d8869c4fb-wqgzn\" (UID: \"ecbc49dd-c78d-429c-bdbb-da551313a61a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3a7b4-d8869c4fb-wqgzn"
Jul 24 17:22:42.759922 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:42.759893    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftbfn\" (UniqueName: \"kubernetes.io/projected/ecbc49dd-c78d-429c-bdbb-da551313a61a-kube-api-access-ftbfn\") pod \"payload-pre-processing-e2e-inf-b-b3a7b4-d8869c4fb-wqgzn\" (UID: \"ecbc49dd-c78d-429c-bdbb-da551313a61a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3a7b4-d8869c4fb-wqgzn"
Jul 24 17:22:42.760343 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:42.760320    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pnlm\" (UniqueName: \"kubernetes.io/projected/a1308141-b7bf-4023-8a73-3af44251c97e-kube-api-access-2pnlm\") pod \"payload-processing-e2e-inf-b-b3a7b4-5d64fcbfb6-6v88l\" (UID: \"a1308141-b7bf-4023-8a73-3af44251c97e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b3a7b4-5d64fcbfb6-6v88l"
Jul 24 17:22:42.807114 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:42.807035    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a7def-674b49955-546z6"
Jul 24 17:22:42.832058 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:42.832019    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3a7b4-7654ffb88b-rrxh4" event={"ID":"2c05f102-c4b1-400e-b664-6f587afe8cb8","Type":"ContainerStarted","Data":"330f3f1dbd2c330093a2d557b4c23aa73d765f16aa939863849d655c3b75ce64"}
Jul 24 17:22:42.832248 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:42.832068    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3a7b4-7654ffb88b-rrxh4" event={"ID":"2c05f102-c4b1-400e-b664-6f587afe8cb8","Type":"ContainerStarted","Data":"4d9f93f5c0302471d9567a4012bcf68d01c8cd3cd5dd1fce76c2b94f9d114782"}
Jul 24 17:22:42.832248 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:42.832157    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3a7b4-7654ffb88b-rrxh4"
Jul 24 17:22:42.849378 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:42.849217    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3a7b4-d8869c4fb-wqgzn"
Jul 24 17:22:42.861244 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:42.861158    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3a7b4-7654ffb88b-rrxh4" podStartSLOduration=1.4738923609999999 podStartE2EDuration="1.861136746s" totalImagesPullingTime="387.244385ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:22:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:22:42.856781862 +0000 UTC m=+2385.498745747" watchObservedRunningTime="2026-07-24 17:22:42.861136746 +0000 UTC m=+2385.503100631"
Jul 24 17:22:42.886725 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:42.886694    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b3a7b4-5d64fcbfb6-6v88l"
Jul 24 17:22:43.018761 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:43.018730    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b3a7b4-d8869c4fb-wqgzn"]
Jul 24 17:22:43.021137 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:22:43.021089    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecbc49dd_c78d_429c_bdbb_da551313a61a.slice/crio-2879316c102f8dfcd1370339c4a475583996602cc9baf43d17a3eee99c07b9e0 WatchSource:0}: Error finding container 2879316c102f8dfcd1370339c4a475583996602cc9baf43d17a3eee99c07b9e0: Status 404 returned error can't find the container with id 2879316c102f8dfcd1370339c4a475583996602cc9baf43d17a3eee99c07b9e0
Jul 24 17:22:43.062181 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:43.062156    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b3a7b4-5d64fcbfb6-6v88l"]
Jul 24 17:22:43.064092 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:22:43.064066    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1308141_b7bf_4023_8a73_3af44251c97e.slice/crio-a6db25a60d5cd5f0c217c741e760bca55b04ed6364863199b0d5e3fbf3258053 WatchSource:0}: Error finding container a6db25a60d5cd5f0c217c741e760bca55b04ed6364863199b0d5e3fbf3258053: Status 404 returned error can't find the container with id a6db25a60d5cd5f0c217c741e760bca55b04ed6364863199b0d5e3fbf3258053
Jul 24 17:22:43.839284 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:43.839246    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3a7b4-d8869c4fb-wqgzn" event={"ID":"ecbc49dd-c78d-429c-bdbb-da551313a61a","Type":"ContainerStarted","Data":"28fb47d1e70c8ca3f0e540574243a67449d6623cc7e3bb47c8de9d28ecdc3d2b"}
Jul 24 17:22:43.839763 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:43.839305    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3a7b4-d8869c4fb-wqgzn" event={"ID":"ecbc49dd-c78d-429c-bdbb-da551313a61a","Type":"ContainerStarted","Data":"2879316c102f8dfcd1370339c4a475583996602cc9baf43d17a3eee99c07b9e0"}
Jul 24 17:22:43.839763 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:43.839336    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3a7b4-d8869c4fb-wqgzn"
Jul 24 17:22:43.840792 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:43.840763    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b3a7b4-5d64fcbfb6-6v88l" event={"ID":"a1308141-b7bf-4023-8a73-3af44251c97e","Type":"ContainerStarted","Data":"f080b24d61669bac6c2b77b0271a1d910c0d2b00022408c49b3344f53a440c87"}
Jul 24 17:22:43.840899 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:43.840815    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b3a7b4-5d64fcbfb6-6v88l" event={"ID":"a1308141-b7bf-4023-8a73-3af44251c97e","Type":"ContainerStarted","Data":"a6db25a60d5cd5f0c217c741e760bca55b04ed6364863199b0d5e3fbf3258053"}
Jul 24 17:22:43.840967 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:43.840898    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b3a7b4-5d64fcbfb6-6v88l"
Jul 24 17:22:43.858547 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:43.858483    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3a7b4-d8869c4fb-wqgzn" podStartSLOduration=1.858462835 podStartE2EDuration="1.858462835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:22:43.855218636 +0000 UTC m=+2386.497182546" watchObservedRunningTime="2026-07-24 17:22:43.858462835 +0000 UTC m=+2386.500426720"
Jul 24 17:22:43.870677 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:43.870616    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-b3a7b4-5d64fcbfb6-6v88l" podStartSLOduration=1.870595475 podStartE2EDuration="1.870595475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:22:43.870200325 +0000 UTC m=+2386.512164230" watchObservedRunningTime="2026-07-24 17:22:43.870595475 +0000 UTC m=+2386.512559359"
Jul 24 17:22:46.141736 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:46.141699    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:22:46.797473 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:46.797443    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0a7def-778dcd6668-r7pf9"
Jul 24 17:22:46.797626 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:46.797513    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0a7def-66bbdb8ff6-cws6n"
Jul 24 17:22:48.846602 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:48.846573    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3a7b4-7654ffb88b-rrxh4"
Jul 24 17:22:50.184964 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:50.184908    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:22:54.847914 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:54.847881    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b3a7b4-5d64fcbfb6-6v88l"
Jul 24 17:22:54.848347 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:54.847980    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3a7b4-d8869c4fb-wqgzn"
Jul 24 17:22:58.144190 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:58.144156    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ddklx_fb5a9838-1e05-4f86-b46c-760bd7aaa038/console-operator/1.log"
Jul 24 17:22:58.152978 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:22:58.152950    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ddklx_fb5a9838-1e05-4f86-b46c-760bd7aaa038/console-operator/1.log"
Jul 24 17:23:08.029656 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:23:08.029616    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:23:35.082539 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:23:35.082464    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:24:00.985580 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:00.985543    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:24:01.930383 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:01.930350    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:24:12.432883 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:12.432842    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0a7def-674b49955-546z6"]
Jul 24 17:24:12.433373 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:12.433127    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a7def-674b49955-546z6" podUID="634d9ea5-ec24-47a3-bdc8-984d1517af22" containerName="maas-api" containerID="cri-o://d07e23e5162e589986662df8afc73965ec29c23c7a0775bb5da3e5de496d4285" gracePeriod=30
Jul 24 17:24:12.456714 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:12.456683    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:24:12.481915 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:12.481869    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0a7def-66bbdb8ff6-cws6n"]
Jul 24 17:24:12.482160 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:12.482122    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-0a7def-66bbdb8ff6-cws6n" podUID="af12882e-1183-42ff-834f-7643d8f0c2ee" containerName="payload-processing" containerID="cri-o://96bdd92f478929eed070966d3019f644943f0592f53c87a06390f2bebeae60cd" gracePeriod=30
Jul 24 17:24:12.491151 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:12.491120    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0a7def-778dcd6668-r7pf9"]
Jul 24 17:24:12.491473 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:12.491407    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0a7def-778dcd6668-r7pf9" podUID="99d80ffc-aabf-4256-848d-02ac2be61fd9" containerName="payload-pre-processing" containerID="cri-o://ab2ae2e787d4d19a0a6a4a1a34dbb9deed29a8ed6736b1280adb6d336b2a6c43" gracePeriod=30
Jul 24 17:24:12.556611 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:12.556576    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:24:12.609788 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:12.609596    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:24:12.795890 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:12.795861    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0a7def-66bbdb8ff6-cws6n"
Jul 24 17:24:12.798664 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:12.798638    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a7def-674b49955-546z6"
Jul 24 17:24:12.802311 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:12.802291    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0a7def-778dcd6668-r7pf9"
Jul 24 17:24:12.975317 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:12.975215    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrkqs\" (UniqueName: \"kubernetes.io/projected/af12882e-1183-42ff-834f-7643d8f0c2ee-kube-api-access-hrkqs\") pod \"af12882e-1183-42ff-834f-7643d8f0c2ee\" (UID: \"af12882e-1183-42ff-834f-7643d8f0c2ee\") "
Jul 24 17:24:12.975317 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:12.975275    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af12882e-1183-42ff-834f-7643d8f0c2ee-plugins-config-volume\") pod \"af12882e-1183-42ff-834f-7643d8f0c2ee\" (UID: \"af12882e-1183-42ff-834f-7643d8f0c2ee\") "
Jul 24 17:24:12.975317 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:12.975293    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fr4d5\" (UniqueName: \"kubernetes.io/projected/634d9ea5-ec24-47a3-bdc8-984d1517af22-kube-api-access-fr4d5\") pod \"634d9ea5-ec24-47a3-bdc8-984d1517af22\" (UID: \"634d9ea5-ec24-47a3-bdc8-984d1517af22\") "
Jul 24 17:24:12.975317 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:12.975316    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/634d9ea5-ec24-47a3-bdc8-984d1517af22-openshift-service-ca\") pod \"634d9ea5-ec24-47a3-bdc8-984d1517af22\" (UID: \"634d9ea5-ec24-47a3-bdc8-984d1517af22\") "
Jul 24 17:24:12.975669 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:12.975357    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99d80ffc-aabf-4256-848d-02ac2be61fd9-plugins-config-volume\") pod \"99d80ffc-aabf-4256-848d-02ac2be61fd9\" (UID: \"99d80ffc-aabf-4256-848d-02ac2be61fd9\") "
Jul 24 17:24:12.975669 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:12.975376    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x69lp\" (UniqueName: \"kubernetes.io/projected/99d80ffc-aabf-4256-848d-02ac2be61fd9-kube-api-access-x69lp\") pod \"99d80ffc-aabf-4256-848d-02ac2be61fd9\" (UID: \"99d80ffc-aabf-4256-848d-02ac2be61fd9\") "
Jul 24 17:24:12.975669 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:12.975421    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/634d9ea5-ec24-47a3-bdc8-984d1517af22-maas-api-tls\") pod \"634d9ea5-ec24-47a3-bdc8-984d1517af22\" (UID: \"634d9ea5-ec24-47a3-bdc8-984d1517af22\") "
Jul 24 17:24:12.975828 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:12.975696    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af12882e-1183-42ff-834f-7643d8f0c2ee-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "af12882e-1183-42ff-834f-7643d8f0c2ee" (UID: "af12882e-1183-42ff-834f-7643d8f0c2ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:24:12.975828 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:12.975748    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/634d9ea5-ec24-47a3-bdc8-984d1517af22-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "634d9ea5-ec24-47a3-bdc8-984d1517af22" (UID: "634d9ea5-ec24-47a3-bdc8-984d1517af22"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:24:12.975828 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:12.975776    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99d80ffc-aabf-4256-848d-02ac2be61fd9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "99d80ffc-aabf-4256-848d-02ac2be61fd9" (UID: "99d80ffc-aabf-4256-848d-02ac2be61fd9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:24:12.977671 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:12.977644    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d80ffc-aabf-4256-848d-02ac2be61fd9-kube-api-access-x69lp" (OuterVolumeSpecName: "kube-api-access-x69lp") pod "99d80ffc-aabf-4256-848d-02ac2be61fd9" (UID: "99d80ffc-aabf-4256-848d-02ac2be61fd9"). InnerVolumeSpecName "kube-api-access-x69lp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:24:12.977857 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:12.977834    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/634d9ea5-ec24-47a3-bdc8-984d1517af22-kube-api-access-fr4d5" (OuterVolumeSpecName: "kube-api-access-fr4d5") pod "634d9ea5-ec24-47a3-bdc8-984d1517af22" (UID: "634d9ea5-ec24-47a3-bdc8-984d1517af22"). InnerVolumeSpecName "kube-api-access-fr4d5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:24:12.977910 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:12.977831    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/634d9ea5-ec24-47a3-bdc8-984d1517af22-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "634d9ea5-ec24-47a3-bdc8-984d1517af22" (UID: "634d9ea5-ec24-47a3-bdc8-984d1517af22"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:24:12.977910 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:12.977879    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af12882e-1183-42ff-834f-7643d8f0c2ee-kube-api-access-hrkqs" (OuterVolumeSpecName: "kube-api-access-hrkqs") pod "af12882e-1183-42ff-834f-7643d8f0c2ee" (UID: "af12882e-1183-42ff-834f-7643d8f0c2ee"). InnerVolumeSpecName "kube-api-access-hrkqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:24:13.076607 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:13.076571    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrkqs\" (UniqueName: \"kubernetes.io/projected/af12882e-1183-42ff-834f-7643d8f0c2ee-kube-api-access-hrkqs\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:24:13.076607 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:13.076599    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af12882e-1183-42ff-834f-7643d8f0c2ee-plugins-config-volume\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:24:13.076819 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:13.076610    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr4d5\" (UniqueName: \"kubernetes.io/projected/634d9ea5-ec24-47a3-bdc8-984d1517af22-kube-api-access-fr4d5\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:24:13.076819 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:13.076643    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/634d9ea5-ec24-47a3-bdc8-984d1517af22-openshift-service-ca\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:24:13.076819 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:13.076655    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99d80ffc-aabf-4256-848d-02ac2be61fd9-plugins-config-volume\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:24:13.076819 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:13.076664    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x69lp\" (UniqueName: \"kubernetes.io/projected/99d80ffc-aabf-4256-848d-02ac2be61fd9-kube-api-access-x69lp\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:24:13.076819 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:13.076674    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/634d9ea5-ec24-47a3-bdc8-984d1517af22-maas-api-tls\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:24:13.309654 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:13.309556    2563 generic.go:356] "Generic (PLEG): container finished" podID="99d80ffc-aabf-4256-848d-02ac2be61fd9" containerID="ab2ae2e787d4d19a0a6a4a1a34dbb9deed29a8ed6736b1280adb6d336b2a6c43" exitCode=0
Jul 24 17:24:13.309654 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:13.309620    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0a7def-778dcd6668-r7pf9"
Jul 24 17:24:13.309654 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:13.309640    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0a7def-778dcd6668-r7pf9" event={"ID":"99d80ffc-aabf-4256-848d-02ac2be61fd9","Type":"ContainerDied","Data":"ab2ae2e787d4d19a0a6a4a1a34dbb9deed29a8ed6736b1280adb6d336b2a6c43"}
Jul 24 17:24:13.309967 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:13.309730    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0a7def-778dcd6668-r7pf9" event={"ID":"99d80ffc-aabf-4256-848d-02ac2be61fd9","Type":"ContainerDied","Data":"06cf444e3050f15e7f2d7c556e942587388326ba5de771d2b55a51ab1a1d6a35"}
Jul 24 17:24:13.309967 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:13.309755    2563 scope.go:117] "RemoveContainer" containerID="ab2ae2e787d4d19a0a6a4a1a34dbb9deed29a8ed6736b1280adb6d336b2a6c43"
Jul 24 17:24:13.311028 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:13.311002    2563 generic.go:356] "Generic (PLEG): container finished" podID="634d9ea5-ec24-47a3-bdc8-984d1517af22" containerID="d07e23e5162e589986662df8afc73965ec29c23c7a0775bb5da3e5de496d4285" exitCode=0
Jul 24 17:24:13.311149 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:13.311055    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a7def-674b49955-546z6"
Jul 24 17:24:13.311149 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:13.311081    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a7def-674b49955-546z6" event={"ID":"634d9ea5-ec24-47a3-bdc8-984d1517af22","Type":"ContainerDied","Data":"d07e23e5162e589986662df8afc73965ec29c23c7a0775bb5da3e5de496d4285"}
Jul 24 17:24:13.311149 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:13.311117    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a7def-674b49955-546z6" event={"ID":"634d9ea5-ec24-47a3-bdc8-984d1517af22","Type":"ContainerDied","Data":"3974434b15a2304ee1b3e396e83f672b412b687535c84901cc5a3135ec93f7a5"}
Jul 24 17:24:13.312454 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:13.312431    2563 generic.go:356] "Generic (PLEG): container finished" podID="af12882e-1183-42ff-834f-7643d8f0c2ee" containerID="96bdd92f478929eed070966d3019f644943f0592f53c87a06390f2bebeae60cd" exitCode=0
Jul 24 17:24:13.312573 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:13.312492    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0a7def-66bbdb8ff6-cws6n"
Jul 24 17:24:13.312573 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:13.312543    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0a7def-66bbdb8ff6-cws6n" event={"ID":"af12882e-1183-42ff-834f-7643d8f0c2ee","Type":"ContainerDied","Data":"96bdd92f478929eed070966d3019f644943f0592f53c87a06390f2bebeae60cd"}
Jul 24 17:24:13.312694 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:13.312597    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0a7def-66bbdb8ff6-cws6n" event={"ID":"af12882e-1183-42ff-834f-7643d8f0c2ee","Type":"ContainerDied","Data":"ff63f4518416259db56b6baebd812f81dcf7f527af650f12867d38040bc5cb90"}
Jul 24 17:24:13.319462 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:13.319438    2563 scope.go:117] "RemoveContainer" containerID="ab2ae2e787d4d19a0a6a4a1a34dbb9deed29a8ed6736b1280adb6d336b2a6c43"
Jul 24 17:24:13.319966 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:24:13.319913    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab2ae2e787d4d19a0a6a4a1a34dbb9deed29a8ed6736b1280adb6d336b2a6c43\": container with ID starting with ab2ae2e787d4d19a0a6a4a1a34dbb9deed29a8ed6736b1280adb6d336b2a6c43 not found: ID does not exist" containerID="ab2ae2e787d4d19a0a6a4a1a34dbb9deed29a8ed6736b1280adb6d336b2a6c43"
Jul 24 17:24:13.320052 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:13.319974    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab2ae2e787d4d19a0a6a4a1a34dbb9deed29a8ed6736b1280adb6d336b2a6c43"} err="failed to get container status \"ab2ae2e787d4d19a0a6a4a1a34dbb9deed29a8ed6736b1280adb6d336b2a6c43\": rpc error: code = NotFound desc = could not find container \"ab2ae2e787d4d19a0a6a4a1a34dbb9deed29a8ed6736b1280adb6d336b2a6c43\": container with ID starting with ab2ae2e787d4d19a0a6a4a1a34dbb9deed29a8ed6736b1280adb6d336b2a6c43 not found: ID does not exist"
Jul 24 17:24:13.320052 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:13.319994    2563 scope.go:117] "RemoveContainer" containerID="d07e23e5162e589986662df8afc73965ec29c23c7a0775bb5da3e5de496d4285"
Jul 24 17:24:13.328684 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:13.328665    2563 scope.go:117] "RemoveContainer" containerID="d07e23e5162e589986662df8afc73965ec29c23c7a0775bb5da3e5de496d4285"
Jul 24 17:24:13.328961 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:24:13.328920    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d07e23e5162e589986662df8afc73965ec29c23c7a0775bb5da3e5de496d4285\": container with ID starting with d07e23e5162e589986662df8afc73965ec29c23c7a0775bb5da3e5de496d4285 not found: ID does not exist" containerID="d07e23e5162e589986662df8afc73965ec29c23c7a0775bb5da3e5de496d4285"
Jul 24 17:24:13.329024 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:13.328971    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d07e23e5162e589986662df8afc73965ec29c23c7a0775bb5da3e5de496d4285"} err="failed to get container status \"d07e23e5162e589986662df8afc73965ec29c23c7a0775bb5da3e5de496d4285\": rpc error: code = NotFound desc = could not find container \"d07e23e5162e589986662df8afc73965ec29c23c7a0775bb5da3e5de496d4285\": container with ID starting with d07e23e5162e589986662df8afc73965ec29c23c7a0775bb5da3e5de496d4285 not found: ID does not exist"
Jul 24 17:24:13.329024 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:13.328993    2563 scope.go:117] "RemoveContainer" containerID="96bdd92f478929eed070966d3019f644943f0592f53c87a06390f2bebeae60cd"
Jul 24 17:24:13.335916 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:13.335894    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0a7def-778dcd6668-r7pf9"]
Jul 24 17:24:13.337157 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:13.337142    2563 scope.go:117] "RemoveContainer" containerID="96bdd92f478929eed070966d3019f644943f0592f53c87a06390f2bebeae60cd"
Jul 24 17:24:13.337507 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:24:13.337482    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96bdd92f478929eed070966d3019f644943f0592f53c87a06390f2bebeae60cd\": container with ID starting with 96bdd92f478929eed070966d3019f644943f0592f53c87a06390f2bebeae60cd not found: ID does not exist" containerID="96bdd92f478929eed070966d3019f644943f0592f53c87a06390f2bebeae60cd"
Jul 24 17:24:13.337647 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:13.337516    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96bdd92f478929eed070966d3019f644943f0592f53c87a06390f2bebeae60cd"} err="failed to get container status \"96bdd92f478929eed070966d3019f644943f0592f53c87a06390f2bebeae60cd\": rpc error: code = NotFound desc = could not find container \"96bdd92f478929eed070966d3019f644943f0592f53c87a06390f2bebeae60cd\": container with ID starting with 96bdd92f478929eed070966d3019f644943f0592f53c87a06390f2bebeae60cd not found: ID does not exist"
Jul 24 17:24:13.340085 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:13.340062    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0a7def-778dcd6668-r7pf9"]
Jul 24 17:24:13.357085 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:13.357049    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0a7def-674b49955-546z6"]
Jul 24 17:24:13.361633 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:13.361605    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0a7def-674b49955-546z6"]
Jul 24 17:24:13.371349 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:13.371317    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0a7def-66bbdb8ff6-cws6n"]
Jul 24 17:24:13.375324 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:13.375300    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0a7def-66bbdb8ff6-cws6n"]
Jul 24 17:24:14.002892 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:14.002856    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="634d9ea5-ec24-47a3-bdc8-984d1517af22" path="/var/lib/kubelet/pods/634d9ea5-ec24-47a3-bdc8-984d1517af22/volumes"
Jul 24 17:24:14.003318 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:14.003241    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99d80ffc-aabf-4256-848d-02ac2be61fd9" path="/var/lib/kubelet/pods/99d80ffc-aabf-4256-848d-02ac2be61fd9/volumes"
Jul 24 17:24:14.003551 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:14.003538    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af12882e-1183-42ff-834f-7643d8f0c2ee" path="/var/lib/kubelet/pods/af12882e-1183-42ff-834f-7643d8f0c2ee/volumes"
Jul 24 17:24:17.150609 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:17.150577    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:24:17.482311 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:17.482225    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:24:23.229956 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:23.229896    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:24:23.282430 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:23.282391    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:24:23.536177 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:23.536089    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:24:23.727526 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:23.727491    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:24:34.212956 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:34.212907    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b3a7b4-7654ffb88b-rrxh4"]
Jul 24 17:24:34.213765 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:34.213728    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3a7b4-7654ffb88b-rrxh4" podUID="2c05f102-c4b1-400e-b664-6f587afe8cb8" containerName="maas-api" containerID="cri-o://330f3f1dbd2c330093a2d557b4c23aa73d765f16aa939863849d655c3b75ce64" gracePeriod=30
Jul 24 17:24:34.244222 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:34.244191    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:24:34.271160 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:34.271130    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b3a7b4-5d64fcbfb6-6v88l"]
Jul 24 17:24:34.271382 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:34.271356    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-b3a7b4-5d64fcbfb6-6v88l" podUID="a1308141-b7bf-4023-8a73-3af44251c97e" containerName="payload-processing" containerID="cri-o://f080b24d61669bac6c2b77b0271a1d910c0d2b00022408c49b3344f53a440c87" gracePeriod=30
Jul 24 17:24:34.273658 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:34.273629    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b3a7b4-d8869c4fb-wqgzn"]
Jul 24 17:24:34.273874 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:34.273846    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3a7b4-d8869c4fb-wqgzn" podUID="ecbc49dd-c78d-429c-bdbb-da551313a61a" containerName="payload-pre-processing" containerID="cri-o://28fb47d1e70c8ca3f0e540574243a67449d6623cc7e3bb47c8de9d28ecdc3d2b" gracePeriod=30
Jul 24 17:24:34.343905 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:34.343880    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:24:34.424121 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:34.424080    2563 generic.go:356] "Generic (PLEG): container finished" podID="a1308141-b7bf-4023-8a73-3af44251c97e" containerID="f080b24d61669bac6c2b77b0271a1d910c0d2b00022408c49b3344f53a440c87" exitCode=0
Jul 24 17:24:34.424302 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:34.424157    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b3a7b4-5d64fcbfb6-6v88l" event={"ID":"a1308141-b7bf-4023-8a73-3af44251c97e","Type":"ContainerDied","Data":"f080b24d61669bac6c2b77b0271a1d910c0d2b00022408c49b3344f53a440c87"}
Jul 24 17:24:34.426156 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:34.426123    2563 generic.go:356] "Generic (PLEG): container finished" podID="ecbc49dd-c78d-429c-bdbb-da551313a61a" containerID="28fb47d1e70c8ca3f0e540574243a67449d6623cc7e3bb47c8de9d28ecdc3d2b" exitCode=0
Jul 24 17:24:34.426301 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:34.426190    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3a7b4-d8869c4fb-wqgzn" event={"ID":"ecbc49dd-c78d-429c-bdbb-da551313a61a","Type":"ContainerDied","Data":"28fb47d1e70c8ca3f0e540574243a67449d6623cc7e3bb47c8de9d28ecdc3d2b"}
Jul 24 17:24:34.445448 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:34.445418    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:24:34.569201 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:34.569143    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b3a7b4-5d64fcbfb6-6v88l"
Jul 24 17:24:34.573372 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:34.573350    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3a7b4-d8869c4fb-wqgzn"
Jul 24 17:24:34.665229 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:34.665205    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3a7b4-7654ffb88b-rrxh4"
Jul 24 17:24:34.665475 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:34.665458    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecbc49dd-c78d-429c-bdbb-da551313a61a-plugins-config-volume\") pod \"ecbc49dd-c78d-429c-bdbb-da551313a61a\" (UID: \"ecbc49dd-c78d-429c-bdbb-da551313a61a\") "
Jul 24 17:24:34.665562 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:34.665492    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2pnlm\" (UniqueName: \"kubernetes.io/projected/a1308141-b7bf-4023-8a73-3af44251c97e-kube-api-access-2pnlm\") pod \"a1308141-b7bf-4023-8a73-3af44251c97e\" (UID: \"a1308141-b7bf-4023-8a73-3af44251c97e\") "
Jul 24 17:24:34.665624 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:34.665564    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1308141-b7bf-4023-8a73-3af44251c97e-plugins-config-volume\") pod \"a1308141-b7bf-4023-8a73-3af44251c97e\" (UID: \"a1308141-b7bf-4023-8a73-3af44251c97e\") "
Jul 24 17:24:34.665624 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:34.665601    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftbfn\" (UniqueName: \"kubernetes.io/projected/ecbc49dd-c78d-429c-bdbb-da551313a61a-kube-api-access-ftbfn\") pod \"ecbc49dd-c78d-429c-bdbb-da551313a61a\" (UID: \"ecbc49dd-c78d-429c-bdbb-da551313a61a\") "
Jul 24 17:24:34.665967 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:34.665925    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecbc49dd-c78d-429c-bdbb-da551313a61a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ecbc49dd-c78d-429c-bdbb-da551313a61a" (UID: "ecbc49dd-c78d-429c-bdbb-da551313a61a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:24:34.666056 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:34.665993    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1308141-b7bf-4023-8a73-3af44251c97e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a1308141-b7bf-4023-8a73-3af44251c97e" (UID: "a1308141-b7bf-4023-8a73-3af44251c97e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:24:34.667526 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:34.667496    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1308141-b7bf-4023-8a73-3af44251c97e-kube-api-access-2pnlm" (OuterVolumeSpecName: "kube-api-access-2pnlm") pod "a1308141-b7bf-4023-8a73-3af44251c97e" (UID: "a1308141-b7bf-4023-8a73-3af44251c97e"). InnerVolumeSpecName "kube-api-access-2pnlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:24:34.667602 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:34.667583    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecbc49dd-c78d-429c-bdbb-da551313a61a-kube-api-access-ftbfn" (OuterVolumeSpecName: "kube-api-access-ftbfn") pod "ecbc49dd-c78d-429c-bdbb-da551313a61a" (UID: "ecbc49dd-c78d-429c-bdbb-da551313a61a"). InnerVolumeSpecName "kube-api-access-ftbfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:24:34.766928 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:34.766840    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c05f102-c4b1-400e-b664-6f587afe8cb8-maas-api-tls\") pod \"2c05f102-c4b1-400e-b664-6f587afe8cb8\" (UID: \"2c05f102-c4b1-400e-b664-6f587afe8cb8\") "
Jul 24 17:24:34.766928 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:34.766916    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqxfn\" (UniqueName: \"kubernetes.io/projected/2c05f102-c4b1-400e-b664-6f587afe8cb8-kube-api-access-mqxfn\") pod \"2c05f102-c4b1-400e-b664-6f587afe8cb8\" (UID: \"2c05f102-c4b1-400e-b664-6f587afe8cb8\") "
Jul 24 17:24:34.767248 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:34.766992    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c05f102-c4b1-400e-b664-6f587afe8cb8-openshift-service-ca\") pod \"2c05f102-c4b1-400e-b664-6f587afe8cb8\" (UID: \"2c05f102-c4b1-400e-b664-6f587afe8cb8\") "
Jul 24 17:24:34.767248 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:34.767156    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1308141-b7bf-4023-8a73-3af44251c97e-plugins-config-volume\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:24:34.767248 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:34.767167    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftbfn\" (UniqueName: \"kubernetes.io/projected/ecbc49dd-c78d-429c-bdbb-da551313a61a-kube-api-access-ftbfn\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:24:34.767248 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:34.767176    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecbc49dd-c78d-429c-bdbb-da551313a61a-plugins-config-volume\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:24:34.767248 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:34.767189    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pnlm\" (UniqueName: \"kubernetes.io/projected/a1308141-b7bf-4023-8a73-3af44251c97e-kube-api-access-2pnlm\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:24:34.767449 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:34.767376    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c05f102-c4b1-400e-b664-6f587afe8cb8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2c05f102-c4b1-400e-b664-6f587afe8cb8" (UID: "2c05f102-c4b1-400e-b664-6f587afe8cb8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:24:34.769007 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:34.768985    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c05f102-c4b1-400e-b664-6f587afe8cb8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2c05f102-c4b1-400e-b664-6f587afe8cb8" (UID: "2c05f102-c4b1-400e-b664-6f587afe8cb8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:24:34.769094 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:34.769074    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c05f102-c4b1-400e-b664-6f587afe8cb8-kube-api-access-mqxfn" (OuterVolumeSpecName: "kube-api-access-mqxfn") pod "2c05f102-c4b1-400e-b664-6f587afe8cb8" (UID: "2c05f102-c4b1-400e-b664-6f587afe8cb8"). InnerVolumeSpecName "kube-api-access-mqxfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:24:34.867566 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:34.867545    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c05f102-c4b1-400e-b664-6f587afe8cb8-maas-api-tls\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:24:34.867566 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:34.867565    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqxfn\" (UniqueName: \"kubernetes.io/projected/2c05f102-c4b1-400e-b664-6f587afe8cb8-kube-api-access-mqxfn\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:24:34.867743 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:34.867576    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c05f102-c4b1-400e-b664-6f587afe8cb8-openshift-service-ca\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:24:35.432023 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:35.431995    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3a7b4-d8869c4fb-wqgzn"
Jul 24 17:24:35.432023 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:35.432021    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3a7b4-d8869c4fb-wqgzn" event={"ID":"ecbc49dd-c78d-429c-bdbb-da551313a61a","Type":"ContainerDied","Data":"2879316c102f8dfcd1370339c4a475583996602cc9baf43d17a3eee99c07b9e0"}
Jul 24 17:24:35.432545 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:35.432083    2563 scope.go:117] "RemoveContainer" containerID="28fb47d1e70c8ca3f0e540574243a67449d6623cc7e3bb47c8de9d28ecdc3d2b"
Jul 24 17:24:35.433332 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:35.433316    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b3a7b4-5d64fcbfb6-6v88l"
Jul 24 17:24:35.433404 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:35.433352    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b3a7b4-5d64fcbfb6-6v88l" event={"ID":"a1308141-b7bf-4023-8a73-3af44251c97e","Type":"ContainerDied","Data":"a6db25a60d5cd5f0c217c741e760bca55b04ed6364863199b0d5e3fbf3258053"}
Jul 24 17:24:35.434722 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:35.434697    2563 generic.go:356] "Generic (PLEG): container finished" podID="2c05f102-c4b1-400e-b664-6f587afe8cb8" containerID="330f3f1dbd2c330093a2d557b4c23aa73d765f16aa939863849d655c3b75ce64" exitCode=0
Jul 24 17:24:35.434808 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:35.434741    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3a7b4-7654ffb88b-rrxh4"
Jul 24 17:24:35.434808 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:35.434752    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3a7b4-7654ffb88b-rrxh4" event={"ID":"2c05f102-c4b1-400e-b664-6f587afe8cb8","Type":"ContainerDied","Data":"330f3f1dbd2c330093a2d557b4c23aa73d765f16aa939863849d655c3b75ce64"}
Jul 24 17:24:35.434808 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:35.434801    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3a7b4-7654ffb88b-rrxh4" event={"ID":"2c05f102-c4b1-400e-b664-6f587afe8cb8","Type":"ContainerDied","Data":"4d9f93f5c0302471d9567a4012bcf68d01c8cd3cd5dd1fce76c2b94f9d114782"}
Jul 24 17:24:35.447355 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:35.442719    2563 scope.go:117] "RemoveContainer" containerID="f080b24d61669bac6c2b77b0271a1d910c0d2b00022408c49b3344f53a440c87"
Jul 24 17:24:35.457485 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:35.457465    2563 scope.go:117] "RemoveContainer" containerID="330f3f1dbd2c330093a2d557b4c23aa73d765f16aa939863849d655c3b75ce64"
Jul 24 17:24:35.465810 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:35.465789    2563 scope.go:117] "RemoveContainer" containerID="330f3f1dbd2c330093a2d557b4c23aa73d765f16aa939863849d655c3b75ce64"
Jul 24 17:24:35.466097 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:24:35.466074    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"330f3f1dbd2c330093a2d557b4c23aa73d765f16aa939863849d655c3b75ce64\": container with ID starting with 330f3f1dbd2c330093a2d557b4c23aa73d765f16aa939863849d655c3b75ce64 not found: ID does not exist" containerID="330f3f1dbd2c330093a2d557b4c23aa73d765f16aa939863849d655c3b75ce64"
Jul 24 17:24:35.466187 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:35.466102    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"330f3f1dbd2c330093a2d557b4c23aa73d765f16aa939863849d655c3b75ce64"} err="failed to get container status \"330f3f1dbd2c330093a2d557b4c23aa73d765f16aa939863849d655c3b75ce64\": rpc error: code = NotFound desc = could not find container \"330f3f1dbd2c330093a2d557b4c23aa73d765f16aa939863849d655c3b75ce64\": container with ID starting with 330f3f1dbd2c330093a2d557b4c23aa73d765f16aa939863849d655c3b75ce64 not found: ID does not exist"
Jul 24 17:24:35.467868 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:35.467850    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b3a7b4-5d64fcbfb6-6v88l"]
Jul 24 17:24:35.469706 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:35.469685    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b3a7b4-5d64fcbfb6-6v88l"]
Jul 24 17:24:35.480728 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:35.480709    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b3a7b4-d8869c4fb-wqgzn"]
Jul 24 17:24:35.484363 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:35.484344    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b3a7b4-d8869c4fb-wqgzn"]
Jul 24 17:24:35.495030 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:35.495009    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b3a7b4-7654ffb88b-rrxh4"]
Jul 24 17:24:35.500523 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:35.500505    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b3a7b4-7654ffb88b-rrxh4"]
Jul 24 17:24:36.002910 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:36.002879    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c05f102-c4b1-400e-b664-6f587afe8cb8" path="/var/lib/kubelet/pods/2c05f102-c4b1-400e-b664-6f587afe8cb8/volumes"
Jul 24 17:24:36.003298 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:36.003282    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1308141-b7bf-4023-8a73-3af44251c97e" path="/var/lib/kubelet/pods/a1308141-b7bf-4023-8a73-3af44251c97e/volumes"
Jul 24 17:24:36.003595 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:36.003584    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecbc49dd-c78d-429c-bdbb-da551313a61a" path="/var/lib/kubelet/pods/ecbc49dd-c78d-429c-bdbb-da551313a61a/volumes"
Jul 24 17:24:38.897544 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:38.897514    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:24:39.286516 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:39.286429    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:24:45.060495 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:45.060461    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:24:45.122814 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:45.122779    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:24:45.320198 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:45.320119    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:24:45.546499 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:45.546462    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:24:45.899971 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:45.899908    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:24:46.201813 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:46.201738    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:24:46.336333 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:46.336298    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:24:49.443923 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:49.443888    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:24:49.553898 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:49.553860    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:24:49.617728 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:49.617693    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c87733-87f8d45b6-vgztw"]
Jul 24 17:24:49.618172 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:49.618153    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c05f102-c4b1-400e-b664-6f587afe8cb8" containerName="maas-api"
Jul 24 17:24:49.618254 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:49.618175    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c05f102-c4b1-400e-b664-6f587afe8cb8" containerName="maas-api"
Jul 24 17:24:49.618254 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:49.618189    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecbc49dd-c78d-429c-bdbb-da551313a61a" containerName="payload-pre-processing"
Jul 24 17:24:49.618254 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:49.618197    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecbc49dd-c78d-429c-bdbb-da551313a61a" containerName="payload-pre-processing"
Jul 24 17:24:49.618254 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:49.618219    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1308141-b7bf-4023-8a73-3af44251c97e" containerName="payload-processing"
Jul 24 17:24:49.618254 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:49.618228    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1308141-b7bf-4023-8a73-3af44251c97e" containerName="payload-processing"
Jul 24 17:24:49.618254 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:49.618241    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af12882e-1183-42ff-834f-7643d8f0c2ee" containerName="payload-processing"
Jul 24 17:24:49.618254 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:49.618248    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="af12882e-1183-42ff-834f-7643d8f0c2ee" containerName="payload-processing"
Jul 24 17:24:49.618254 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:49.618254    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99d80ffc-aabf-4256-848d-02ac2be61fd9" containerName="payload-pre-processing"
Jul 24 17:24:49.618689 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:49.618260    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="99d80ffc-aabf-4256-848d-02ac2be61fd9" containerName="payload-pre-processing"
Jul 24 17:24:49.618689 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:49.618271    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="634d9ea5-ec24-47a3-bdc8-984d1517af22" containerName="maas-api"
Jul 24 17:24:49.618689 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:49.618276    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="634d9ea5-ec24-47a3-bdc8-984d1517af22" containerName="maas-api"
Jul 24 17:24:49.618689 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:49.618369    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="634d9ea5-ec24-47a3-bdc8-984d1517af22" containerName="maas-api"
Jul 24 17:24:49.618689 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:49.618378    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="af12882e-1183-42ff-834f-7643d8f0c2ee" containerName="payload-processing"
Jul 24 17:24:49.618689 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:49.618385    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c05f102-c4b1-400e-b664-6f587afe8cb8" containerName="maas-api"
Jul 24 17:24:49.618689 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:49.618392    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecbc49dd-c78d-429c-bdbb-da551313a61a" containerName="payload-pre-processing"
Jul 24 17:24:49.618689 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:49.618398    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1308141-b7bf-4023-8a73-3af44251c97e" containerName="payload-processing"
Jul 24 17:24:49.618689 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:49.618411    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="99d80ffc-aabf-4256-848d-02ac2be61fd9" containerName="payload-pre-processing"
Jul 24 17:24:49.622904 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:49.622881    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c87733-87f8d45b6-vgztw"
Jul 24 17:24:49.625672 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:49.625645    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c87733-serving-cert\""
Jul 24 17:24:49.632763 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:49.632743    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c87733-87f8d45b6-vgztw"]
Jul 24 17:24:49.647221 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:49.647192    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:24:49.700081 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:49.700008    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54m44\" (UniqueName: \"kubernetes.io/projected/e546ab8a-6078-4a57-835f-12021c383f24-kube-api-access-54m44\") pod \"maas-api-e2e-api-a-c87733-87f8d45b6-vgztw\" (UID: \"e546ab8a-6078-4a57-835f-12021c383f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c87733-87f8d45b6-vgztw"
Jul 24 17:24:49.700081 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:49.700055    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e546ab8a-6078-4a57-835f-12021c383f24-maas-api-tls\") pod \"maas-api-e2e-api-a-c87733-87f8d45b6-vgztw\" (UID: \"e546ab8a-6078-4a57-835f-12021c383f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c87733-87f8d45b6-vgztw"
Jul 24 17:24:49.700246 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:49.700177    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e546ab8a-6078-4a57-835f-12021c383f24-openshift-service-ca\") pod \"maas-api-e2e-api-a-c87733-87f8d45b6-vgztw\" (UID: \"e546ab8a-6078-4a57-835f-12021c383f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c87733-87f8d45b6-vgztw"
Jul 24 17:24:49.789871 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:49.789843    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:24:49.801070 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:49.801032    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e546ab8a-6078-4a57-835f-12021c383f24-openshift-service-ca\") pod \"maas-api-e2e-api-a-c87733-87f8d45b6-vgztw\" (UID: \"e546ab8a-6078-4a57-835f-12021c383f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c87733-87f8d45b6-vgztw"
Jul 24 17:24:49.801209 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:49.801101    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54m44\" (UniqueName: \"kubernetes.io/projected/e546ab8a-6078-4a57-835f-12021c383f24-kube-api-access-54m44\") pod \"maas-api-e2e-api-a-c87733-87f8d45b6-vgztw\" (UID: \"e546ab8a-6078-4a57-835f-12021c383f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c87733-87f8d45b6-vgztw"
Jul 24 17:24:49.801209 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:49.801140    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e546ab8a-6078-4a57-835f-12021c383f24-maas-api-tls\") pod \"maas-api-e2e-api-a-c87733-87f8d45b6-vgztw\" (UID: \"e546ab8a-6078-4a57-835f-12021c383f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c87733-87f8d45b6-vgztw"
Jul 24 17:24:49.801671 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:49.801651    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e546ab8a-6078-4a57-835f-12021c383f24-openshift-service-ca\") pod \"maas-api-e2e-api-a-c87733-87f8d45b6-vgztw\" (UID: \"e546ab8a-6078-4a57-835f-12021c383f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c87733-87f8d45b6-vgztw"
Jul 24 17:24:49.803538 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:49.803509    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e546ab8a-6078-4a57-835f-12021c383f24-maas-api-tls\") pod \"maas-api-e2e-api-a-c87733-87f8d45b6-vgztw\" (UID: \"e546ab8a-6078-4a57-835f-12021c383f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c87733-87f8d45b6-vgztw"
Jul 24 17:24:49.809878 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:49.809855    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54m44\" (UniqueName: \"kubernetes.io/projected/e546ab8a-6078-4a57-835f-12021c383f24-kube-api-access-54m44\") pod \"maas-api-e2e-api-a-c87733-87f8d45b6-vgztw\" (UID: \"e546ab8a-6078-4a57-835f-12021c383f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c87733-87f8d45b6-vgztw"
Jul 24 17:24:49.933675 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:49.933645    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c87733-87f8d45b6-vgztw"
Jul 24 17:24:50.128187 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:50.128115    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:24:50.232435 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:50.232380    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:24:50.241240 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:50.241172    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c87733-bf8dd6684-cjr9w"]
Jul 24 17:24:50.245865 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:50.245847    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c87733-bf8dd6684-cjr9w"
Jul 24 17:24:50.248897 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:50.248877    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c87733\""
Jul 24 17:24:50.249012 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:50.248882    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c87733-dockercfg-rprwh\""
Jul 24 17:24:50.263471 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:50.263400    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c87733-bf8dd6684-cjr9w"]
Jul 24 17:24:50.266335 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:50.266313    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c87733-9d47c5958-ltvbp"]
Jul 24 17:24:50.269848 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:50.269823    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c87733-9d47c5958-ltvbp"
Jul 24 17:24:50.285468 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:50.285445    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c87733-9d47c5958-ltvbp"]
Jul 24 17:24:50.291606 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:50.291584    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c87733-87f8d45b6-vgztw"]
Jul 24 17:24:50.294062 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:24:50.294035    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode546ab8a_6078_4a57_835f_12021c383f24.slice/crio-ff98e8cf36fd964532a6c57b42c682da3a1dcbc5a2187964c503ac5d05e16d6f WatchSource:0}: Error finding container ff98e8cf36fd964532a6c57b42c682da3a1dcbc5a2187964c503ac5d05e16d6f: Status 404 returned error can't find the container with id ff98e8cf36fd964532a6c57b42c682da3a1dcbc5a2187964c503ac5d05e16d6f
Jul 24 17:24:50.295371 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:50.295354    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 17:24:50.406813 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:50.406781    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-596gt\" (UniqueName: \"kubernetes.io/projected/fe1f3a66-3f28-418e-ae20-1d1bf066319e-kube-api-access-596gt\") pod \"payload-processing-e2e-api-a-c87733-9d47c5958-ltvbp\" (UID: \"fe1f3a66-3f28-418e-ae20-1d1bf066319e\") " pod="openshift-ingress/payload-processing-e2e-api-a-c87733-9d47c5958-ltvbp"
Jul 24 17:24:50.407030 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:50.406820    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stqsr\" (UniqueName: \"kubernetes.io/projected/6d85ecd6-1a1b-4609-be64-4c0d5a3f6526-kube-api-access-stqsr\") pod \"payload-pre-processing-e2e-api-a-c87733-bf8dd6684-cjr9w\" (UID: \"6d85ecd6-1a1b-4609-be64-4c0d5a3f6526\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c87733-bf8dd6684-cjr9w"
Jul 24 17:24:50.407030 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:50.406844    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d85ecd6-1a1b-4609-be64-4c0d5a3f6526-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c87733-bf8dd6684-cjr9w\" (UID: \"6d85ecd6-1a1b-4609-be64-4c0d5a3f6526\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c87733-bf8dd6684-cjr9w"
Jul 24 17:24:50.407030 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:50.406867    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe1f3a66-3f28-418e-ae20-1d1bf066319e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c87733-9d47c5958-ltvbp\" (UID: \"fe1f3a66-3f28-418e-ae20-1d1bf066319e\") " pod="openshift-ingress/payload-processing-e2e-api-a-c87733-9d47c5958-ltvbp"
Jul 24 17:24:50.507644 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:50.507555    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stqsr\" (UniqueName: \"kubernetes.io/projected/6d85ecd6-1a1b-4609-be64-4c0d5a3f6526-kube-api-access-stqsr\") pod \"payload-pre-processing-e2e-api-a-c87733-bf8dd6684-cjr9w\" (UID: \"6d85ecd6-1a1b-4609-be64-4c0d5a3f6526\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c87733-bf8dd6684-cjr9w"
Jul 24 17:24:50.507644 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:50.507598    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d85ecd6-1a1b-4609-be64-4c0d5a3f6526-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c87733-bf8dd6684-cjr9w\" (UID: \"6d85ecd6-1a1b-4609-be64-4c0d5a3f6526\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c87733-bf8dd6684-cjr9w"
Jul 24 17:24:50.508222 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:50.507718    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe1f3a66-3f28-418e-ae20-1d1bf066319e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c87733-9d47c5958-ltvbp\" (UID: \"fe1f3a66-3f28-418e-ae20-1d1bf066319e\") " pod="openshift-ingress/payload-processing-e2e-api-a-c87733-9d47c5958-ltvbp"
Jul 24 17:24:50.508222 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:50.507844    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-596gt\" (UniqueName: \"kubernetes.io/projected/fe1f3a66-3f28-418e-ae20-1d1bf066319e-kube-api-access-596gt\") pod \"payload-processing-e2e-api-a-c87733-9d47c5958-ltvbp\" (UID: \"fe1f3a66-3f28-418e-ae20-1d1bf066319e\") " pod="openshift-ingress/payload-processing-e2e-api-a-c87733-9d47c5958-ltvbp"
Jul 24 17:24:50.508457 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:50.508436    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe1f3a66-3f28-418e-ae20-1d1bf066319e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c87733-9d47c5958-ltvbp\" (UID: \"fe1f3a66-3f28-418e-ae20-1d1bf066319e\") " pod="openshift-ingress/payload-processing-e2e-api-a-c87733-9d47c5958-ltvbp"
Jul 24 17:24:50.508457 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:50.508445    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d85ecd6-1a1b-4609-be64-4c0d5a3f6526-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c87733-bf8dd6684-cjr9w\" (UID: \"6d85ecd6-1a1b-4609-be64-4c0d5a3f6526\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c87733-bf8dd6684-cjr9w"
Jul 24 17:24:50.515089 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:50.515066    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c87733-87f8d45b6-vgztw" event={"ID":"e546ab8a-6078-4a57-835f-12021c383f24","Type":"ContainerStarted","Data":"ff98e8cf36fd964532a6c57b42c682da3a1dcbc5a2187964c503ac5d05e16d6f"}
Jul 24 17:24:50.515260 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:50.515243    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stqsr\" (UniqueName: \"kubernetes.io/projected/6d85ecd6-1a1b-4609-be64-4c0d5a3f6526-kube-api-access-stqsr\") pod \"payload-pre-processing-e2e-api-a-c87733-bf8dd6684-cjr9w\" (UID: \"6d85ecd6-1a1b-4609-be64-4c0d5a3f6526\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c87733-bf8dd6684-cjr9w"
Jul 24 17:24:50.515576 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:50.515560    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-596gt\" (UniqueName: \"kubernetes.io/projected/fe1f3a66-3f28-418e-ae20-1d1bf066319e-kube-api-access-596gt\") pod \"payload-processing-e2e-api-a-c87733-9d47c5958-ltvbp\" (UID: \"fe1f3a66-3f28-418e-ae20-1d1bf066319e\") " pod="openshift-ingress/payload-processing-e2e-api-a-c87733-9d47c5958-ltvbp"
Jul 24 17:24:50.556530 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:50.556505    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c87733-bf8dd6684-cjr9w"
Jul 24 17:24:50.580275 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:50.580248    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c87733-9d47c5958-ltvbp"
Jul 24 17:24:50.710653 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:50.710620    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c87733-bf8dd6684-cjr9w"]
Jul 24 17:24:50.713970 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:24:50.713921    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d85ecd6_1a1b_4609_be64_4c0d5a3f6526.slice/crio-5d330370608d10fc8cb01d3b038aa027f2062ebfe36bcdac4adef82ab38785cb WatchSource:0}: Error finding container 5d330370608d10fc8cb01d3b038aa027f2062ebfe36bcdac4adef82ab38785cb: Status 404 returned error can't find the container with id 5d330370608d10fc8cb01d3b038aa027f2062ebfe36bcdac4adef82ab38785cb
Jul 24 17:24:50.934529 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:50.934440    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c87733-9d47c5958-ltvbp"]
Jul 24 17:24:50.936730 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:24:50.936687    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe1f3a66_3f28_418e_ae20_1d1bf066319e.slice/crio-a0957eda94eb348e67a372d2acb5b782ad434ec30a306a2879603a6deef1d207 WatchSource:0}: Error finding container a0957eda94eb348e67a372d2acb5b782ad434ec30a306a2879603a6deef1d207: Status 404 returned error can't find the container with id a0957eda94eb348e67a372d2acb5b782ad434ec30a306a2879603a6deef1d207
Jul 24 17:24:51.521800 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:51.521766    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c87733-9d47c5958-ltvbp" event={"ID":"fe1f3a66-3f28-418e-ae20-1d1bf066319e","Type":"ContainerStarted","Data":"8dde57d6569ddf187e8b1ebd54d3dd41bc81ee72ba0e00b0c63fa0e4b5273005"}
Jul 24 17:24:51.522288 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:51.521820    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c87733-9d47c5958-ltvbp" event={"ID":"fe1f3a66-3f28-418e-ae20-1d1bf066319e","Type":"ContainerStarted","Data":"a0957eda94eb348e67a372d2acb5b782ad434ec30a306a2879603a6deef1d207"}
Jul 24 17:24:51.522288 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:51.521878    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-c87733-9d47c5958-ltvbp"
Jul 24 17:24:51.523400 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:51.523373    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c87733-87f8d45b6-vgztw" event={"ID":"e546ab8a-6078-4a57-835f-12021c383f24","Type":"ContainerStarted","Data":"e422a51efe1475acd1e8c47dffbdcd9b23aefb26f64687ca8f161e0fa244b94e"}
Jul 24 17:24:51.523530 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:51.523414    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c87733-87f8d45b6-vgztw"
Jul 24 17:24:51.524871 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:51.524848    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c87733-bf8dd6684-cjr9w" event={"ID":"6d85ecd6-1a1b-4609-be64-4c0d5a3f6526","Type":"ContainerStarted","Data":"5b27925b65aa95371a1b95c290796edd370fdb5e940620ab0fccfaea11064299"}
Jul 24 17:24:51.525022 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:51.524894    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c87733-bf8dd6684-cjr9w" event={"ID":"6d85ecd6-1a1b-4609-be64-4c0d5a3f6526","Type":"ContainerStarted","Data":"5d330370608d10fc8cb01d3b038aa027f2062ebfe36bcdac4adef82ab38785cb"}
Jul 24 17:24:51.525022 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:51.524924    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c87733-bf8dd6684-cjr9w"
Jul 24 17:24:51.539715 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:51.539673    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-c87733-9d47c5958-ltvbp" podStartSLOduration=1.5396598460000002 podStartE2EDuration="1.539659846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:24:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:24:51.537911091 +0000 UTC m=+2514.179874976" watchObservedRunningTime="2026-07-24 17:24:51.539659846 +0000 UTC m=+2514.181623708"
Jul 24 17:24:51.561735 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:51.561678    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c87733-87f8d45b6-vgztw" podStartSLOduration=2.15620576 podStartE2EDuration="2.56165964s" totalImagesPullingTime="405.45388ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:24:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:24:51.558808258 +0000 UTC m=+2514.200772167" watchObservedRunningTime="2026-07-24 17:24:51.56165964 +0000 UTC m=+2514.203623525"
Jul 24 17:24:51.575424 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:51.575376    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c87733-bf8dd6684-cjr9w" podStartSLOduration=1.57536347 podStartE2EDuration="1.57536347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:24:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:24:51.573589349 +0000 UTC m=+2514.215553247" watchObservedRunningTime="2026-07-24 17:24:51.57536347 +0000 UTC m=+2514.217327353"
Jul 24 17:24:53.921474 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:53.921445    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:24:54.393285 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:54.393254    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:24:54.802676 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:54.802584    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:24:54.944407 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:54.944366    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:24:57.473510 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:57.473481    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:24:57.535428 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:57.535398    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c87733-87f8d45b6-vgztw"
Jul 24 17:24:57.545963 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:57.545908    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-04ba18-857cdc455d-4wxjt"]
Jul 24 17:24:57.548915 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:57.548893    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:24:57.550023 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:57.549977    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-04ba18-857cdc455d-4wxjt"
Jul 24 17:24:57.552375 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:57.552350    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-04ba18-serving-cert\""
Jul 24 17:24:57.559888 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:57.559487    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-04ba18-857cdc455d-4wxjt"]
Jul 24 17:24:57.617952 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:57.617904    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:24:57.674571 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:57.674543    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e84a08eb-ad2c-46ce-a1d5-721d80b9e130-maas-api-tls\") pod \"maas-api-e2e-api-b-04ba18-857cdc455d-4wxjt\" (UID: \"e84a08eb-ad2c-46ce-a1d5-721d80b9e130\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-04ba18-857cdc455d-4wxjt"
Jul 24 17:24:57.674744 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:57.674652    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gj65\" (UniqueName: \"kubernetes.io/projected/e84a08eb-ad2c-46ce-a1d5-721d80b9e130-kube-api-access-2gj65\") pod \"maas-api-e2e-api-b-04ba18-857cdc455d-4wxjt\" (UID: \"e84a08eb-ad2c-46ce-a1d5-721d80b9e130\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-04ba18-857cdc455d-4wxjt"
Jul 24 17:24:57.674744 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:57.674697    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e84a08eb-ad2c-46ce-a1d5-721d80b9e130-openshift-service-ca\") pod \"maas-api-e2e-api-b-04ba18-857cdc455d-4wxjt\" (UID: \"e84a08eb-ad2c-46ce-a1d5-721d80b9e130\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-04ba18-857cdc455d-4wxjt"
Jul 24 17:24:57.775772 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:57.775679    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e84a08eb-ad2c-46ce-a1d5-721d80b9e130-maas-api-tls\") pod \"maas-api-e2e-api-b-04ba18-857cdc455d-4wxjt\" (UID: \"e84a08eb-ad2c-46ce-a1d5-721d80b9e130\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-04ba18-857cdc455d-4wxjt"
Jul 24 17:24:57.775772 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:57.775740    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gj65\" (UniqueName: \"kubernetes.io/projected/e84a08eb-ad2c-46ce-a1d5-721d80b9e130-kube-api-access-2gj65\") pod \"maas-api-e2e-api-b-04ba18-857cdc455d-4wxjt\" (UID: \"e84a08eb-ad2c-46ce-a1d5-721d80b9e130\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-04ba18-857cdc455d-4wxjt"
Jul 24 17:24:57.775772 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:57.775771    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e84a08eb-ad2c-46ce-a1d5-721d80b9e130-openshift-service-ca\") pod \"maas-api-e2e-api-b-04ba18-857cdc455d-4wxjt\" (UID: \"e84a08eb-ad2c-46ce-a1d5-721d80b9e130\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-04ba18-857cdc455d-4wxjt"
Jul 24 17:24:57.776409 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:57.776391    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e84a08eb-ad2c-46ce-a1d5-721d80b9e130-openshift-service-ca\") pod \"maas-api-e2e-api-b-04ba18-857cdc455d-4wxjt\" (UID: \"e84a08eb-ad2c-46ce-a1d5-721d80b9e130\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-04ba18-857cdc455d-4wxjt"
Jul 24 17:24:57.778292 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:57.778274    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-04ba18-serving-cert\""
Jul 24 17:24:57.785178 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:57.785155    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gj65\" (UniqueName: \"kubernetes.io/projected/e84a08eb-ad2c-46ce-a1d5-721d80b9e130-kube-api-access-2gj65\") pod \"maas-api-e2e-api-b-04ba18-857cdc455d-4wxjt\" (UID: \"e84a08eb-ad2c-46ce-a1d5-721d80b9e130\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-04ba18-857cdc455d-4wxjt"
Jul 24 17:24:57.788886 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:57.788863    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e84a08eb-ad2c-46ce-a1d5-721d80b9e130-maas-api-tls\") pod \"maas-api-e2e-api-b-04ba18-857cdc455d-4wxjt\" (UID: \"e84a08eb-ad2c-46ce-a1d5-721d80b9e130\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-04ba18-857cdc455d-4wxjt"
Jul 24 17:24:57.812342 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:57.812320    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:24:57.865178 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:57.865147    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-04ba18-857cdc455d-4wxjt"
Jul 24 17:24:58.012209 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:58.011169    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-04ba18-857cdc455d-4wxjt"]
Jul 24 17:24:58.191448 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:58.191420    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-04ba18-5f966d7b4d-lb6c2"]
Jul 24 17:24:58.196300 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:58.196284    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-04ba18-5f966d7b4d-lb6c2"
Jul 24 17:24:58.200430 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:58.200408    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-04ba18-dockercfg-nbnzl\""
Jul 24 17:24:58.200749 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:58.200733    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-04ba18\""
Jul 24 17:24:58.213440 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:58.213419    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-04ba18-5596b6fdf8-x8lmd"]
Jul 24 17:24:58.216822 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:58.216803    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-04ba18-5f966d7b4d-lb6c2"]
Jul 24 17:24:58.216919 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:58.216905    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-04ba18-5596b6fdf8-x8lmd"
Jul 24 17:24:58.234732 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:58.234701    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-04ba18-5596b6fdf8-x8lmd"]
Jul 24 17:24:58.381191 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:58.381164    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8gwg\" (UniqueName: \"kubernetes.io/projected/103e7629-5bff-4486-be12-8ea3fc7bf09f-kube-api-access-z8gwg\") pod \"payload-processing-e2e-api-b-04ba18-5596b6fdf8-x8lmd\" (UID: \"103e7629-5bff-4486-be12-8ea3fc7bf09f\") " pod="openshift-ingress/payload-processing-e2e-api-b-04ba18-5596b6fdf8-x8lmd"
Jul 24 17:24:58.381318 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:58.381229    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a7b6b92-9dd4-4465-81c0-c0904662bf43-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-04ba18-5f966d7b4d-lb6c2\" (UID: \"0a7b6b92-9dd4-4465-81c0-c0904662bf43\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-04ba18-5f966d7b4d-lb6c2"
Jul 24 17:24:58.381388 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:58.381309    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/103e7629-5bff-4486-be12-8ea3fc7bf09f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-04ba18-5596b6fdf8-x8lmd\" (UID: \"103e7629-5bff-4486-be12-8ea3fc7bf09f\") " pod="openshift-ingress/payload-processing-e2e-api-b-04ba18-5596b6fdf8-x8lmd"
Jul 24 17:24:58.381388 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:58.381352    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkx9g\" (UniqueName: \"kubernetes.io/projected/0a7b6b92-9dd4-4465-81c0-c0904662bf43-kube-api-access-zkx9g\") pod \"payload-pre-processing-e2e-api-b-04ba18-5f966d7b4d-lb6c2\" (UID: \"0a7b6b92-9dd4-4465-81c0-c0904662bf43\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-04ba18-5f966d7b4d-lb6c2"
Jul 24 17:24:58.481799 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:58.481767    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8gwg\" (UniqueName: \"kubernetes.io/projected/103e7629-5bff-4486-be12-8ea3fc7bf09f-kube-api-access-z8gwg\") pod \"payload-processing-e2e-api-b-04ba18-5596b6fdf8-x8lmd\" (UID: \"103e7629-5bff-4486-be12-8ea3fc7bf09f\") " pod="openshift-ingress/payload-processing-e2e-api-b-04ba18-5596b6fdf8-x8lmd"
Jul 24 17:24:58.482198 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:58.481860    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a7b6b92-9dd4-4465-81c0-c0904662bf43-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-04ba18-5f966d7b4d-lb6c2\" (UID: \"0a7b6b92-9dd4-4465-81c0-c0904662bf43\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-04ba18-5f966d7b4d-lb6c2"
Jul 24 17:24:58.482198 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:58.481895    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/103e7629-5bff-4486-be12-8ea3fc7bf09f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-04ba18-5596b6fdf8-x8lmd\" (UID: \"103e7629-5bff-4486-be12-8ea3fc7bf09f\") " pod="openshift-ingress/payload-processing-e2e-api-b-04ba18-5596b6fdf8-x8lmd"
Jul 24 17:24:58.482198 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:58.481921    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkx9g\" (UniqueName: \"kubernetes.io/projected/0a7b6b92-9dd4-4465-81c0-c0904662bf43-kube-api-access-zkx9g\") pod \"payload-pre-processing-e2e-api-b-04ba18-5f966d7b4d-lb6c2\" (UID: \"0a7b6b92-9dd4-4465-81c0-c0904662bf43\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-04ba18-5f966d7b4d-lb6c2"
Jul 24 17:24:58.482820 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:58.482769    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a7b6b92-9dd4-4465-81c0-c0904662bf43-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-04ba18-5f966d7b4d-lb6c2\" (UID: \"0a7b6b92-9dd4-4465-81c0-c0904662bf43\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-04ba18-5f966d7b4d-lb6c2"
Jul 24 17:24:58.482977 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:58.482891    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/103e7629-5bff-4486-be12-8ea3fc7bf09f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-04ba18-5596b6fdf8-x8lmd\" (UID: \"103e7629-5bff-4486-be12-8ea3fc7bf09f\") " pod="openshift-ingress/payload-processing-e2e-api-b-04ba18-5596b6fdf8-x8lmd"
Jul 24 17:24:58.489977 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:58.489950    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8gwg\" (UniqueName: \"kubernetes.io/projected/103e7629-5bff-4486-be12-8ea3fc7bf09f-kube-api-access-z8gwg\") pod \"payload-processing-e2e-api-b-04ba18-5596b6fdf8-x8lmd\" (UID: \"103e7629-5bff-4486-be12-8ea3fc7bf09f\") " pod="openshift-ingress/payload-processing-e2e-api-b-04ba18-5596b6fdf8-x8lmd"
Jul 24 17:24:58.490079 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:58.489983    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkx9g\" (UniqueName: \"kubernetes.io/projected/0a7b6b92-9dd4-4465-81c0-c0904662bf43-kube-api-access-zkx9g\") pod \"payload-pre-processing-e2e-api-b-04ba18-5f966d7b4d-lb6c2\" (UID: \"0a7b6b92-9dd4-4465-81c0-c0904662bf43\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-04ba18-5f966d7b4d-lb6c2"
Jul 24 17:24:58.506903 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:58.506881    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-04ba18-5f966d7b4d-lb6c2"
Jul 24 17:24:58.526974 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:58.526910    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-04ba18-5596b6fdf8-x8lmd"
Jul 24 17:24:58.574609 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:58.574563    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-04ba18-857cdc455d-4wxjt" event={"ID":"e84a08eb-ad2c-46ce-a1d5-721d80b9e130","Type":"ContainerStarted","Data":"521c45f32710783e1a34ddef410ca069242e93e2f0c30f3cbfbb422e0a105142"}
Jul 24 17:24:58.574775 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:58.574626    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-04ba18-857cdc455d-4wxjt" event={"ID":"e84a08eb-ad2c-46ce-a1d5-721d80b9e130","Type":"ContainerStarted","Data":"4420f67f7ff498071660dd3100539535152b79331fa0ae4c774582f83cf4096e"}
Jul 24 17:24:58.574775 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:58.574676    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-04ba18-857cdc455d-4wxjt"
Jul 24 17:24:58.598659 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:58.598600    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-04ba18-857cdc455d-4wxjt" podStartSLOduration=1.234871807 podStartE2EDuration="1.598564903s" totalImagesPullingTime="363.693096ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:24:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:24:58.597885906 +0000 UTC m=+2521.239849791" watchObservedRunningTime="2026-07-24 17:24:58.598564903 +0000 UTC m=+2521.240528787"
Jul 24 17:24:58.656861 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:58.656705    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-04ba18-5f966d7b4d-lb6c2"]
Jul 24 17:24:58.659674 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:24:58.659627    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a7b6b92_9dd4_4465_81c0_c0904662bf43.slice/crio-68a3a8d79f319ee21683f70c4c4d4c5cae571e3a20ca7b010fe8080f4cc595e2 WatchSource:0}: Error finding container 68a3a8d79f319ee21683f70c4c4d4c5cae571e3a20ca7b010fe8080f4cc595e2: Status 404 returned error can't find the container with id 68a3a8d79f319ee21683f70c4c4d4c5cae571e3a20ca7b010fe8080f4cc595e2
Jul 24 17:24:58.888980 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:58.888959    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-04ba18-5596b6fdf8-x8lmd"]
Jul 24 17:24:58.891008 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:24:58.890978    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod103e7629_5bff_4486_be12_8ea3fc7bf09f.slice/crio-49b4ff3904174ad464928120238ae3289f5b9fea3cac1f8d4962bebfb5f159f1 WatchSource:0}: Error finding container 49b4ff3904174ad464928120238ae3289f5b9fea3cac1f8d4962bebfb5f159f1: Status 404 returned error can't find the container with id 49b4ff3904174ad464928120238ae3289f5b9fea3cac1f8d4962bebfb5f159f1
Jul 24 17:24:59.012501 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:59.012466    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:24:59.583007 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:59.582972    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-04ba18-5596b6fdf8-x8lmd" event={"ID":"103e7629-5bff-4486-be12-8ea3fc7bf09f","Type":"ContainerStarted","Data":"1c09b3064e30b9f3080444c5d1ab20d53d33f818e4de21ed689271c02c860ee9"}
Jul 24 17:24:59.583501 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:59.583030    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-04ba18-5596b6fdf8-x8lmd" event={"ID":"103e7629-5bff-4486-be12-8ea3fc7bf09f","Type":"ContainerStarted","Data":"49b4ff3904174ad464928120238ae3289f5b9fea3cac1f8d4962bebfb5f159f1"}
Jul 24 17:24:59.583501 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:59.583055    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-04ba18-5596b6fdf8-x8lmd"
Jul 24 17:24:59.584513 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:59.584487    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-04ba18-5f966d7b4d-lb6c2" event={"ID":"0a7b6b92-9dd4-4465-81c0-c0904662bf43","Type":"ContainerStarted","Data":"5e4b4922e3c350e84d574a60a442b292f3ad409d05140c162e9aa5eb95db31d8"}
Jul 24 17:24:59.584619 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:59.584538    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-04ba18-5f966d7b4d-lb6c2" event={"ID":"0a7b6b92-9dd4-4465-81c0-c0904662bf43","Type":"ContainerStarted","Data":"68a3a8d79f319ee21683f70c4c4d4c5cae571e3a20ca7b010fe8080f4cc595e2"}
Jul 24 17:24:59.584716 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:59.584696    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-04ba18-5f966d7b4d-lb6c2"
Jul 24 17:24:59.598171 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:59.598119    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-04ba18-5596b6fdf8-x8lmd" podStartSLOduration=1.5981028849999999 podStartE2EDuration="1.598102885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:24:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:24:59.597471277 +0000 UTC m=+2522.239435174" watchObservedRunningTime="2026-07-24 17:24:59.598102885 +0000 UTC m=+2522.240066770"
Jul 24 17:24:59.614732 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:24:59.614685    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-04ba18-5f966d7b4d-lb6c2" podStartSLOduration=1.614667694 podStartE2EDuration="1.614667694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:24:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:24:59.61166361 +0000 UTC m=+2522.253627495" watchObservedRunningTime="2026-07-24 17:24:59.614667694 +0000 UTC m=+2522.256631579"
Jul 24 17:25:01.973147 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:01.973093    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:25:02.532009 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:02.531980    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c87733-bf8dd6684-cjr9w"
Jul 24 17:25:02.532281 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:02.532257    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-c87733-9d47c5958-ltvbp"
Jul 24 17:25:04.590125 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:04.590095    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-04ba18-857cdc455d-4wxjt"
Jul 24 17:25:08.094784 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:08.094741    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:25:08.154329 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:08.154300    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:25:10.591536 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:10.591496    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-04ba18-5596b6fdf8-x8lmd"
Jul 24 17:25:10.591920 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:10.591576    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-04ba18-5f966d7b4d-lb6c2"
Jul 24 17:25:15.783022 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:15.782988    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:25:25.665254 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:25.665216    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c87733-87f8d45b6-vgztw"]
Jul 24 17:25:25.665660 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:25.665455    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c87733-87f8d45b6-vgztw" podUID="e546ab8a-6078-4a57-835f-12021c383f24" containerName="maas-api" containerID="cri-o://e422a51efe1475acd1e8c47dffbdcd9b23aefb26f64687ca8f161e0fa244b94e" gracePeriod=30
Jul 24 17:25:25.696102 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:25.696073    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:25:25.722089 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:25.722061    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c87733-9d47c5958-ltvbp"]
Jul 24 17:25:25.722317 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:25.722283    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-c87733-9d47c5958-ltvbp" podUID="fe1f3a66-3f28-418e-ae20-1d1bf066319e" containerName="payload-processing" containerID="cri-o://8dde57d6569ddf187e8b1ebd54d3dd41bc81ee72ba0e00b0c63fa0e4b5273005" gracePeriod=30
Jul 24 17:25:25.727749 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:25.727720    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c87733-bf8dd6684-cjr9w"]
Jul 24 17:25:25.728170 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:25.728118    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c87733-bf8dd6684-cjr9w" podUID="6d85ecd6-1a1b-4609-be64-4c0d5a3f6526" containerName="payload-pre-processing" containerID="cri-o://5b27925b65aa95371a1b95c290796edd370fdb5e940620ab0fccfaea11064299" gracePeriod=30
Jul 24 17:25:25.764957 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:25.764906    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:25:25.865742 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:25.865694    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:25:26.005384 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.005360    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c87733-9d47c5958-ltvbp"
Jul 24 17:25:26.018423 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.018395    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c87733-87f8d45b6-vgztw"
Jul 24 17:25:26.026641 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.026567    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c87733-bf8dd6684-cjr9w"
Jul 24 17:25:26.127667 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.127627    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stqsr\" (UniqueName: \"kubernetes.io/projected/6d85ecd6-1a1b-4609-be64-4c0d5a3f6526-kube-api-access-stqsr\") pod \"6d85ecd6-1a1b-4609-be64-4c0d5a3f6526\" (UID: \"6d85ecd6-1a1b-4609-be64-4c0d5a3f6526\") "
Jul 24 17:25:26.127843 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.127693    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e546ab8a-6078-4a57-835f-12021c383f24-maas-api-tls\") pod \"e546ab8a-6078-4a57-835f-12021c383f24\" (UID: \"e546ab8a-6078-4a57-835f-12021c383f24\") "
Jul 24 17:25:26.127910 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.127873    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54m44\" (UniqueName: \"kubernetes.io/projected/e546ab8a-6078-4a57-835f-12021c383f24-kube-api-access-54m44\") pod \"e546ab8a-6078-4a57-835f-12021c383f24\" (UID: \"e546ab8a-6078-4a57-835f-12021c383f24\") "
Jul 24 17:25:26.128012 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.127958    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d85ecd6-1a1b-4609-be64-4c0d5a3f6526-plugins-config-volume\") pod \"6d85ecd6-1a1b-4609-be64-4c0d5a3f6526\" (UID: \"6d85ecd6-1a1b-4609-be64-4c0d5a3f6526\") "
Jul 24 17:25:26.128012 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.128009    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-596gt\" (UniqueName: \"kubernetes.io/projected/fe1f3a66-3f28-418e-ae20-1d1bf066319e-kube-api-access-596gt\") pod \"fe1f3a66-3f28-418e-ae20-1d1bf066319e\" (UID: \"fe1f3a66-3f28-418e-ae20-1d1bf066319e\") "
Jul 24 17:25:26.128125 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.128070    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e546ab8a-6078-4a57-835f-12021c383f24-openshift-service-ca\") pod \"e546ab8a-6078-4a57-835f-12021c383f24\" (UID: \"e546ab8a-6078-4a57-835f-12021c383f24\") "
Jul 24 17:25:26.128125 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.128099    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe1f3a66-3f28-418e-ae20-1d1bf066319e-plugins-config-volume\") pod \"fe1f3a66-3f28-418e-ae20-1d1bf066319e\" (UID: \"fe1f3a66-3f28-418e-ae20-1d1bf066319e\") "
Jul 24 17:25:26.128514 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.128482    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d85ecd6-1a1b-4609-be64-4c0d5a3f6526-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6d85ecd6-1a1b-4609-be64-4c0d5a3f6526" (UID: "6d85ecd6-1a1b-4609-be64-4c0d5a3f6526"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:25:26.128621 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.128506    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e546ab8a-6078-4a57-835f-12021c383f24-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e546ab8a-6078-4a57-835f-12021c383f24" (UID: "e546ab8a-6078-4a57-835f-12021c383f24"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:25:26.128621 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.128557    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe1f3a66-3f28-418e-ae20-1d1bf066319e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fe1f3a66-3f28-418e-ae20-1d1bf066319e" (UID: "fe1f3a66-3f28-418e-ae20-1d1bf066319e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:25:26.129908 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.129886    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e546ab8a-6078-4a57-835f-12021c383f24-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e546ab8a-6078-4a57-835f-12021c383f24" (UID: "e546ab8a-6078-4a57-835f-12021c383f24"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:25:26.130060 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.129960    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d85ecd6-1a1b-4609-be64-4c0d5a3f6526-kube-api-access-stqsr" (OuterVolumeSpecName: "kube-api-access-stqsr") pod "6d85ecd6-1a1b-4609-be64-4c0d5a3f6526" (UID: "6d85ecd6-1a1b-4609-be64-4c0d5a3f6526"). InnerVolumeSpecName "kube-api-access-stqsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:25:26.130289 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.130269    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e546ab8a-6078-4a57-835f-12021c383f24-kube-api-access-54m44" (OuterVolumeSpecName: "kube-api-access-54m44") pod "e546ab8a-6078-4a57-835f-12021c383f24" (UID: "e546ab8a-6078-4a57-835f-12021c383f24"). InnerVolumeSpecName "kube-api-access-54m44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:25:26.130369 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.130354    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe1f3a66-3f28-418e-ae20-1d1bf066319e-kube-api-access-596gt" (OuterVolumeSpecName: "kube-api-access-596gt") pod "fe1f3a66-3f28-418e-ae20-1d1bf066319e" (UID: "fe1f3a66-3f28-418e-ae20-1d1bf066319e"). InnerVolumeSpecName "kube-api-access-596gt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:25:26.229747 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.229664    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54m44\" (UniqueName: \"kubernetes.io/projected/e546ab8a-6078-4a57-835f-12021c383f24-kube-api-access-54m44\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:25:26.229747 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.229692    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d85ecd6-1a1b-4609-be64-4c0d5a3f6526-plugins-config-volume\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:25:26.229747 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.229701    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-596gt\" (UniqueName: \"kubernetes.io/projected/fe1f3a66-3f28-418e-ae20-1d1bf066319e-kube-api-access-596gt\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:25:26.229747 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.229709    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e546ab8a-6078-4a57-835f-12021c383f24-openshift-service-ca\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:25:26.229747 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.229718    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe1f3a66-3f28-418e-ae20-1d1bf066319e-plugins-config-volume\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:25:26.229747 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.229725    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stqsr\" (UniqueName: \"kubernetes.io/projected/6d85ecd6-1a1b-4609-be64-4c0d5a3f6526-kube-api-access-stqsr\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:25:26.229747 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.229735    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e546ab8a-6078-4a57-835f-12021c383f24-maas-api-tls\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:25:26.732923 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.732888    2563 generic.go:356] "Generic (PLEG): container finished" podID="fe1f3a66-3f28-418e-ae20-1d1bf066319e" containerID="8dde57d6569ddf187e8b1ebd54d3dd41bc81ee72ba0e00b0c63fa0e4b5273005" exitCode=0
Jul 24 17:25:26.733356 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.732966    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c87733-9d47c5958-ltvbp"
Jul 24 17:25:26.733356 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.732976    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c87733-9d47c5958-ltvbp" event={"ID":"fe1f3a66-3f28-418e-ae20-1d1bf066319e","Type":"ContainerDied","Data":"8dde57d6569ddf187e8b1ebd54d3dd41bc81ee72ba0e00b0c63fa0e4b5273005"}
Jul 24 17:25:26.733356 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.733058    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c87733-9d47c5958-ltvbp" event={"ID":"fe1f3a66-3f28-418e-ae20-1d1bf066319e","Type":"ContainerDied","Data":"a0957eda94eb348e67a372d2acb5b782ad434ec30a306a2879603a6deef1d207"}
Jul 24 17:25:26.733356 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.733085    2563 scope.go:117] "RemoveContainer" containerID="8dde57d6569ddf187e8b1ebd54d3dd41bc81ee72ba0e00b0c63fa0e4b5273005"
Jul 24 17:25:26.734331 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.734284    2563 generic.go:356] "Generic (PLEG): container finished" podID="e546ab8a-6078-4a57-835f-12021c383f24" containerID="e422a51efe1475acd1e8c47dffbdcd9b23aefb26f64687ca8f161e0fa244b94e" exitCode=0
Jul 24 17:25:26.734411 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.734343    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c87733-87f8d45b6-vgztw"
Jul 24 17:25:26.734411 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.734366    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c87733-87f8d45b6-vgztw" event={"ID":"e546ab8a-6078-4a57-835f-12021c383f24","Type":"ContainerDied","Data":"e422a51efe1475acd1e8c47dffbdcd9b23aefb26f64687ca8f161e0fa244b94e"}
Jul 24 17:25:26.734524 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.734444    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c87733-87f8d45b6-vgztw" event={"ID":"e546ab8a-6078-4a57-835f-12021c383f24","Type":"ContainerDied","Data":"ff98e8cf36fd964532a6c57b42c682da3a1dcbc5a2187964c503ac5d05e16d6f"}
Jul 24 17:25:26.735664 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.735638    2563 generic.go:356] "Generic (PLEG): container finished" podID="6d85ecd6-1a1b-4609-be64-4c0d5a3f6526" containerID="5b27925b65aa95371a1b95c290796edd370fdb5e940620ab0fccfaea11064299" exitCode=0
Jul 24 17:25:26.735784 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.735687    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c87733-bf8dd6684-cjr9w"
Jul 24 17:25:26.735784 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.735753    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c87733-bf8dd6684-cjr9w" event={"ID":"6d85ecd6-1a1b-4609-be64-4c0d5a3f6526","Type":"ContainerDied","Data":"5b27925b65aa95371a1b95c290796edd370fdb5e940620ab0fccfaea11064299"}
Jul 24 17:25:26.735890 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.735787    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c87733-bf8dd6684-cjr9w" event={"ID":"6d85ecd6-1a1b-4609-be64-4c0d5a3f6526","Type":"ContainerDied","Data":"5d330370608d10fc8cb01d3b038aa027f2062ebfe36bcdac4adef82ab38785cb"}
Jul 24 17:25:26.744066 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.744036    2563 scope.go:117] "RemoveContainer" containerID="8dde57d6569ddf187e8b1ebd54d3dd41bc81ee72ba0e00b0c63fa0e4b5273005"
Jul 24 17:25:26.744418 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:25:26.744391    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dde57d6569ddf187e8b1ebd54d3dd41bc81ee72ba0e00b0c63fa0e4b5273005\": container with ID starting with 8dde57d6569ddf187e8b1ebd54d3dd41bc81ee72ba0e00b0c63fa0e4b5273005 not found: ID does not exist" containerID="8dde57d6569ddf187e8b1ebd54d3dd41bc81ee72ba0e00b0c63fa0e4b5273005"
Jul 24 17:25:26.744526 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.744422    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dde57d6569ddf187e8b1ebd54d3dd41bc81ee72ba0e00b0c63fa0e4b5273005"} err="failed to get container status \"8dde57d6569ddf187e8b1ebd54d3dd41bc81ee72ba0e00b0c63fa0e4b5273005\": rpc error: code = NotFound desc = could not find container \"8dde57d6569ddf187e8b1ebd54d3dd41bc81ee72ba0e00b0c63fa0e4b5273005\": container with ID starting with 8dde57d6569ddf187e8b1ebd54d3dd41bc81ee72ba0e00b0c63fa0e4b5273005 not found: ID does not exist"
Jul 24 17:25:26.744526 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.744445    2563 scope.go:117] "RemoveContainer" containerID="e422a51efe1475acd1e8c47dffbdcd9b23aefb26f64687ca8f161e0fa244b94e"
Jul 24 17:25:26.753324 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.753221    2563 scope.go:117] "RemoveContainer" containerID="e422a51efe1475acd1e8c47dffbdcd9b23aefb26f64687ca8f161e0fa244b94e"
Jul 24 17:25:26.753617 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:25:26.753590    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e422a51efe1475acd1e8c47dffbdcd9b23aefb26f64687ca8f161e0fa244b94e\": container with ID starting with e422a51efe1475acd1e8c47dffbdcd9b23aefb26f64687ca8f161e0fa244b94e not found: ID does not exist" containerID="e422a51efe1475acd1e8c47dffbdcd9b23aefb26f64687ca8f161e0fa244b94e"
Jul 24 17:25:26.753700 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.753625    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e422a51efe1475acd1e8c47dffbdcd9b23aefb26f64687ca8f161e0fa244b94e"} err="failed to get container status \"e422a51efe1475acd1e8c47dffbdcd9b23aefb26f64687ca8f161e0fa244b94e\": rpc error: code = NotFound desc = could not find container \"e422a51efe1475acd1e8c47dffbdcd9b23aefb26f64687ca8f161e0fa244b94e\": container with ID starting with e422a51efe1475acd1e8c47dffbdcd9b23aefb26f64687ca8f161e0fa244b94e not found: ID does not exist"
Jul 24 17:25:26.753700 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.753645    2563 scope.go:117] "RemoveContainer" containerID="5b27925b65aa95371a1b95c290796edd370fdb5e940620ab0fccfaea11064299"
Jul 24 17:25:26.755919 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.755899    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c87733-9d47c5958-ltvbp"]
Jul 24 17:25:26.759622 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.759598    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c87733-9d47c5958-ltvbp"]
Jul 24 17:25:26.761598 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.761581    2563 scope.go:117] "RemoveContainer" containerID="5b27925b65aa95371a1b95c290796edd370fdb5e940620ab0fccfaea11064299"
Jul 24 17:25:26.761827 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:25:26.761810    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b27925b65aa95371a1b95c290796edd370fdb5e940620ab0fccfaea11064299\": container with ID starting with 5b27925b65aa95371a1b95c290796edd370fdb5e940620ab0fccfaea11064299 not found: ID does not exist" containerID="5b27925b65aa95371a1b95c290796edd370fdb5e940620ab0fccfaea11064299"
Jul 24 17:25:26.761872 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.761835    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b27925b65aa95371a1b95c290796edd370fdb5e940620ab0fccfaea11064299"} err="failed to get container status \"5b27925b65aa95371a1b95c290796edd370fdb5e940620ab0fccfaea11064299\": rpc error: code = NotFound desc = could not find container \"5b27925b65aa95371a1b95c290796edd370fdb5e940620ab0fccfaea11064299\": container with ID starting with 5b27925b65aa95371a1b95c290796edd370fdb5e940620ab0fccfaea11064299 not found: ID does not exist"
Jul 24 17:25:26.769688 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.769667    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c87733-87f8d45b6-vgztw"]
Jul 24 17:25:26.774912 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.774893    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c87733-87f8d45b6-vgztw"]
Jul 24 17:25:26.784398 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.784378    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c87733-bf8dd6684-cjr9w"]
Jul 24 17:25:26.790102 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:26.790081    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c87733-bf8dd6684-cjr9w"]
Jul 24 17:25:28.002443 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:28.002397    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d85ecd6-1a1b-4609-be64-4c0d5a3f6526" path="/var/lib/kubelet/pods/6d85ecd6-1a1b-4609-be64-4c0d5a3f6526/volumes"
Jul 24 17:25:28.002985 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:28.002966    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e546ab8a-6078-4a57-835f-12021c383f24" path="/var/lib/kubelet/pods/e546ab8a-6078-4a57-835f-12021c383f24/volumes"
Jul 24 17:25:28.003364 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:28.003346    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe1f3a66-3f28-418e-ae20-1d1bf066319e" path="/var/lib/kubelet/pods/fe1f3a66-3f28-418e-ae20-1d1bf066319e/volumes"
Jul 24 17:25:30.042080 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:30.042048    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:25:30.430102 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:30.430073    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:25:36.167881 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:36.167838    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:25:36.222616 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:36.222586    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:25:36.437087 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:36.437010    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:25:36.652961 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:36.652918    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:25:50.629008 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:50.628974    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:25:52.237180 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.237143    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-04ba18-857cdc455d-4wxjt"]
Jul 24 17:25:52.237580 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.237353    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-04ba18-857cdc455d-4wxjt" podUID="e84a08eb-ad2c-46ce-a1d5-721d80b9e130" containerName="maas-api" containerID="cri-o://521c45f32710783e1a34ddef410ca069242e93e2f0c30f3cbfbb422e0a105142" gracePeriod=30
Jul 24 17:25:52.262298 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.262270    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:25:52.287717 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.287693    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-04ba18-5596b6fdf8-x8lmd"]
Jul 24 17:25:52.287931 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.287910    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-04ba18-5596b6fdf8-x8lmd" podUID="103e7629-5bff-4486-be12-8ea3fc7bf09f" containerName="payload-processing" containerID="cri-o://1c09b3064e30b9f3080444c5d1ab20d53d33f818e4de21ed689271c02c860ee9" gracePeriod=30
Jul 24 17:25:52.293625 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.293598    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-04ba18-5f966d7b4d-lb6c2"]
Jul 24 17:25:52.293836 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.293790    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-04ba18-5f966d7b4d-lb6c2" podUID="0a7b6b92-9dd4-4465-81c0-c0904662bf43" containerName="payload-pre-processing" containerID="cri-o://5e4b4922e3c350e84d574a60a442b292f3ad409d05140c162e9aa5eb95db31d8" gracePeriod=30
Jul 24 17:25:52.365302 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.365278    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:25:52.467201 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.467128    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:25:52.541610 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.541580    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-04ba18-5596b6fdf8-x8lmd"
Jul 24 17:25:52.565289 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.565021    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8gwg\" (UniqueName: \"kubernetes.io/projected/103e7629-5bff-4486-be12-8ea3fc7bf09f-kube-api-access-z8gwg\") pod \"103e7629-5bff-4486-be12-8ea3fc7bf09f\" (UID: \"103e7629-5bff-4486-be12-8ea3fc7bf09f\") "
Jul 24 17:25:52.565289 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.565111    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/103e7629-5bff-4486-be12-8ea3fc7bf09f-plugins-config-volume\") pod \"103e7629-5bff-4486-be12-8ea3fc7bf09f\" (UID: \"103e7629-5bff-4486-be12-8ea3fc7bf09f\") "
Jul 24 17:25:52.565735 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.565699    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/103e7629-5bff-4486-be12-8ea3fc7bf09f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "103e7629-5bff-4486-be12-8ea3fc7bf09f" (UID: "103e7629-5bff-4486-be12-8ea3fc7bf09f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:25:52.568464 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.568432    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103e7629-5bff-4486-be12-8ea3fc7bf09f-kube-api-access-z8gwg" (OuterVolumeSpecName: "kube-api-access-z8gwg") pod "103e7629-5bff-4486-be12-8ea3fc7bf09f" (UID: "103e7629-5bff-4486-be12-8ea3fc7bf09f"). InnerVolumeSpecName "kube-api-access-z8gwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:25:52.591473 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.591439    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-04ba18-5f966d7b4d-lb6c2"
Jul 24 17:25:52.594213 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.594192    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-04ba18-857cdc455d-4wxjt"
Jul 24 17:25:52.666192 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.666162    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a7b6b92-9dd4-4465-81c0-c0904662bf43-plugins-config-volume\") pod \"0a7b6b92-9dd4-4465-81c0-c0904662bf43\" (UID: \"0a7b6b92-9dd4-4465-81c0-c0904662bf43\") "
Jul 24 17:25:52.666352 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.666199    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkx9g\" (UniqueName: \"kubernetes.io/projected/0a7b6b92-9dd4-4465-81c0-c0904662bf43-kube-api-access-zkx9g\") pod \"0a7b6b92-9dd4-4465-81c0-c0904662bf43\" (UID: \"0a7b6b92-9dd4-4465-81c0-c0904662bf43\") "
Jul 24 17:25:52.666352 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.666291    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e84a08eb-ad2c-46ce-a1d5-721d80b9e130-maas-api-tls\") pod \"e84a08eb-ad2c-46ce-a1d5-721d80b9e130\" (UID: \"e84a08eb-ad2c-46ce-a1d5-721d80b9e130\") "
Jul 24 17:25:52.666352 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.666322    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gj65\" (UniqueName: \"kubernetes.io/projected/e84a08eb-ad2c-46ce-a1d5-721d80b9e130-kube-api-access-2gj65\") pod \"e84a08eb-ad2c-46ce-a1d5-721d80b9e130\" (UID: \"e84a08eb-ad2c-46ce-a1d5-721d80b9e130\") "
Jul 24 17:25:52.666519 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.666371    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e84a08eb-ad2c-46ce-a1d5-721d80b9e130-openshift-service-ca\") pod \"e84a08eb-ad2c-46ce-a1d5-721d80b9e130\" (UID: \"e84a08eb-ad2c-46ce-a1d5-721d80b9e130\") "
Jul 24 17:25:52.666614 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.666591    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a7b6b92-9dd4-4465-81c0-c0904662bf43-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0a7b6b92-9dd4-4465-81c0-c0904662bf43" (UID: "0a7b6b92-9dd4-4465-81c0-c0904662bf43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:25:52.666731 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.666707    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a7b6b92-9dd4-4465-81c0-c0904662bf43-plugins-config-volume\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:25:52.666731 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.666733    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8gwg\" (UniqueName: \"kubernetes.io/projected/103e7629-5bff-4486-be12-8ea3fc7bf09f-kube-api-access-z8gwg\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:25:52.666884 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.666748    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/103e7629-5bff-4486-be12-8ea3fc7bf09f-plugins-config-volume\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:25:52.666884 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.666794    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e84a08eb-ad2c-46ce-a1d5-721d80b9e130-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e84a08eb-ad2c-46ce-a1d5-721d80b9e130" (UID: "e84a08eb-ad2c-46ce-a1d5-721d80b9e130"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:25:52.668430 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.668410    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e84a08eb-ad2c-46ce-a1d5-721d80b9e130-kube-api-access-2gj65" (OuterVolumeSpecName: "kube-api-access-2gj65") pod "e84a08eb-ad2c-46ce-a1d5-721d80b9e130" (UID: "e84a08eb-ad2c-46ce-a1d5-721d80b9e130"). InnerVolumeSpecName "kube-api-access-2gj65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:25:52.668796 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.668774    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e84a08eb-ad2c-46ce-a1d5-721d80b9e130-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e84a08eb-ad2c-46ce-a1d5-721d80b9e130" (UID: "e84a08eb-ad2c-46ce-a1d5-721d80b9e130"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:25:52.668875 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.668776    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a7b6b92-9dd4-4465-81c0-c0904662bf43-kube-api-access-zkx9g" (OuterVolumeSpecName: "kube-api-access-zkx9g") pod "0a7b6b92-9dd4-4465-81c0-c0904662bf43" (UID: "0a7b6b92-9dd4-4465-81c0-c0904662bf43"). InnerVolumeSpecName "kube-api-access-zkx9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:25:52.768056 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.767922    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e84a08eb-ad2c-46ce-a1d5-721d80b9e130-openshift-service-ca\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:25:52.768056 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.768000    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkx9g\" (UniqueName: \"kubernetes.io/projected/0a7b6b92-9dd4-4465-81c0-c0904662bf43-kube-api-access-zkx9g\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:25:52.768056 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.768012    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e84a08eb-ad2c-46ce-a1d5-721d80b9e130-maas-api-tls\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:25:52.768056 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.768021    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gj65\" (UniqueName: \"kubernetes.io/projected/e84a08eb-ad2c-46ce-a1d5-721d80b9e130-kube-api-access-2gj65\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:25:52.880887 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.880854    2563 generic.go:356] "Generic (PLEG): container finished" podID="0a7b6b92-9dd4-4465-81c0-c0904662bf43" containerID="5e4b4922e3c350e84d574a60a442b292f3ad409d05140c162e9aa5eb95db31d8" exitCode=0
Jul 24 17:25:52.881078 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.880893    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-04ba18-5f966d7b4d-lb6c2"
Jul 24 17:25:52.881078 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.880959    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-04ba18-5f966d7b4d-lb6c2" event={"ID":"0a7b6b92-9dd4-4465-81c0-c0904662bf43","Type":"ContainerDied","Data":"5e4b4922e3c350e84d574a60a442b292f3ad409d05140c162e9aa5eb95db31d8"}
Jul 24 17:25:52.881078 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.881013    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-04ba18-5f966d7b4d-lb6c2" event={"ID":"0a7b6b92-9dd4-4465-81c0-c0904662bf43","Type":"ContainerDied","Data":"68a3a8d79f319ee21683f70c4c4d4c5cae571e3a20ca7b010fe8080f4cc595e2"}
Jul 24 17:25:52.881078 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.881043    2563 scope.go:117] "RemoveContainer" containerID="5e4b4922e3c350e84d574a60a442b292f3ad409d05140c162e9aa5eb95db31d8"
Jul 24 17:25:52.882267 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.882247    2563 generic.go:356] "Generic (PLEG): container finished" podID="103e7629-5bff-4486-be12-8ea3fc7bf09f" containerID="1c09b3064e30b9f3080444c5d1ab20d53d33f818e4de21ed689271c02c860ee9" exitCode=0
Jul 24 17:25:52.882375 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.882283    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-04ba18-5596b6fdf8-x8lmd"
Jul 24 17:25:52.882375 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.882332    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-04ba18-5596b6fdf8-x8lmd" event={"ID":"103e7629-5bff-4486-be12-8ea3fc7bf09f","Type":"ContainerDied","Data":"1c09b3064e30b9f3080444c5d1ab20d53d33f818e4de21ed689271c02c860ee9"}
Jul 24 17:25:52.882508 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.882384    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-04ba18-5596b6fdf8-x8lmd" event={"ID":"103e7629-5bff-4486-be12-8ea3fc7bf09f","Type":"ContainerDied","Data":"49b4ff3904174ad464928120238ae3289f5b9fea3cac1f8d4962bebfb5f159f1"}
Jul 24 17:25:52.883728 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.883700    2563 generic.go:356] "Generic (PLEG): container finished" podID="e84a08eb-ad2c-46ce-a1d5-721d80b9e130" containerID="521c45f32710783e1a34ddef410ca069242e93e2f0c30f3cbfbb422e0a105142" exitCode=0
Jul 24 17:25:52.883862 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.883758    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-04ba18-857cdc455d-4wxjt"
Jul 24 17:25:52.883862 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.883760    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-04ba18-857cdc455d-4wxjt" event={"ID":"e84a08eb-ad2c-46ce-a1d5-721d80b9e130","Type":"ContainerDied","Data":"521c45f32710783e1a34ddef410ca069242e93e2f0c30f3cbfbb422e0a105142"}
Jul 24 17:25:52.884006 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.883870    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-04ba18-857cdc455d-4wxjt" event={"ID":"e84a08eb-ad2c-46ce-a1d5-721d80b9e130","Type":"ContainerDied","Data":"4420f67f7ff498071660dd3100539535152b79331fa0ae4c774582f83cf4096e"}
Jul 24 17:25:52.890100 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.890080    2563 scope.go:117] "RemoveContainer" containerID="5e4b4922e3c350e84d574a60a442b292f3ad409d05140c162e9aa5eb95db31d8"
Jul 24 17:25:52.890380 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:25:52.890359    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e4b4922e3c350e84d574a60a442b292f3ad409d05140c162e9aa5eb95db31d8\": container with ID starting with 5e4b4922e3c350e84d574a60a442b292f3ad409d05140c162e9aa5eb95db31d8 not found: ID does not exist" containerID="5e4b4922e3c350e84d574a60a442b292f3ad409d05140c162e9aa5eb95db31d8"
Jul 24 17:25:52.890481 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.890387    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e4b4922e3c350e84d574a60a442b292f3ad409d05140c162e9aa5eb95db31d8"} err="failed to get container status \"5e4b4922e3c350e84d574a60a442b292f3ad409d05140c162e9aa5eb95db31d8\": rpc error: code = NotFound desc = could not find container \"5e4b4922e3c350e84d574a60a442b292f3ad409d05140c162e9aa5eb95db31d8\": container with ID starting with 5e4b4922e3c350e84d574a60a442b292f3ad409d05140c162e9aa5eb95db31d8 not found: ID does not exist"
Jul 24 17:25:52.890481 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.890403    2563 scope.go:117] "RemoveContainer" containerID="1c09b3064e30b9f3080444c5d1ab20d53d33f818e4de21ed689271c02c860ee9"
Jul 24 17:25:52.899886 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.899870    2563 scope.go:117] "RemoveContainer" containerID="1c09b3064e30b9f3080444c5d1ab20d53d33f818e4de21ed689271c02c860ee9"
Jul 24 17:25:52.900146 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:25:52.900128    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c09b3064e30b9f3080444c5d1ab20d53d33f818e4de21ed689271c02c860ee9\": container with ID starting with 1c09b3064e30b9f3080444c5d1ab20d53d33f818e4de21ed689271c02c860ee9 not found: ID does not exist" containerID="1c09b3064e30b9f3080444c5d1ab20d53d33f818e4de21ed689271c02c860ee9"
Jul 24 17:25:52.900190 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.900151    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c09b3064e30b9f3080444c5d1ab20d53d33f818e4de21ed689271c02c860ee9"} err="failed to get container status \"1c09b3064e30b9f3080444c5d1ab20d53d33f818e4de21ed689271c02c860ee9\": rpc error: code = NotFound desc = could not find container \"1c09b3064e30b9f3080444c5d1ab20d53d33f818e4de21ed689271c02c860ee9\": container with ID starting with 1c09b3064e30b9f3080444c5d1ab20d53d33f818e4de21ed689271c02c860ee9 not found: ID does not exist"
Jul 24 17:25:52.900190 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.900179    2563 scope.go:117] "RemoveContainer" containerID="521c45f32710783e1a34ddef410ca069242e93e2f0c30f3cbfbb422e0a105142"
Jul 24 17:25:52.907846 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.907821    2563 scope.go:117] "RemoveContainer" containerID="521c45f32710783e1a34ddef410ca069242e93e2f0c30f3cbfbb422e0a105142"
Jul 24 17:25:52.908108 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:25:52.908091    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"521c45f32710783e1a34ddef410ca069242e93e2f0c30f3cbfbb422e0a105142\": container with ID starting with 521c45f32710783e1a34ddef410ca069242e93e2f0c30f3cbfbb422e0a105142 not found: ID does not exist" containerID="521c45f32710783e1a34ddef410ca069242e93e2f0c30f3cbfbb422e0a105142"
Jul 24 17:25:52.908162 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.908115    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"521c45f32710783e1a34ddef410ca069242e93e2f0c30f3cbfbb422e0a105142"} err="failed to get container status \"521c45f32710783e1a34ddef410ca069242e93e2f0c30f3cbfbb422e0a105142\": rpc error: code = NotFound desc = could not find container \"521c45f32710783e1a34ddef410ca069242e93e2f0c30f3cbfbb422e0a105142\": container with ID starting with 521c45f32710783e1a34ddef410ca069242e93e2f0c30f3cbfbb422e0a105142 not found: ID does not exist"
Jul 24 17:25:52.913178 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.913157    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-04ba18-5596b6fdf8-x8lmd"]
Jul 24 17:25:52.917071 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.917054    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-04ba18-5596b6fdf8-x8lmd"]
Jul 24 17:25:52.934802 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.934781    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-04ba18-857cdc455d-4wxjt"]
Jul 24 17:25:52.942487 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.942469    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-04ba18-857cdc455d-4wxjt"]
Jul 24 17:25:52.959526 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.959502    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-04ba18-5f966d7b4d-lb6c2"]
Jul 24 17:25:52.962570 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:52.962547    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-04ba18-5f966d7b4d-lb6c2"]
Jul 24 17:25:54.001699 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:54.001666    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a7b6b92-9dd4-4465-81c0-c0904662bf43" path="/var/lib/kubelet/pods/0a7b6b92-9dd4-4465-81c0-c0904662bf43/volumes"
Jul 24 17:25:54.002090 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:54.002036    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="103e7629-5bff-4486-be12-8ea3fc7bf09f" path="/var/lib/kubelet/pods/103e7629-5bff-4486-be12-8ea3fc7bf09f/volumes"
Jul 24 17:25:54.002379 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:54.002366    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e84a08eb-ad2c-46ce-a1d5-721d80b9e130" path="/var/lib/kubelet/pods/e84a08eb-ad2c-46ce-a1d5-721d80b9e130/volumes"
Jul 24 17:25:56.841520 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:56.841481    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:25:57.302616 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:25:57.302584    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:26:02.928562 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:02.928523    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:26:02.995598 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:02.995570    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:26:03.202552 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:03.202477    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:26:03.424075 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:03.424046    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:26:03.753165 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:03.753125    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:26:04.079459 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:04.079388    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:26:04.242909 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:04.242875    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:26:07.029089 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.029053    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:26:07.113728 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.113696    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:26:07.224283 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.224251    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-de68c7-79ccd89d4f-sxqrz"]
Jul 24 17:26:07.224674 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.224658    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="103e7629-5bff-4486-be12-8ea3fc7bf09f" containerName="payload-processing"
Jul 24 17:26:07.224760 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.224676    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="103e7629-5bff-4486-be12-8ea3fc7bf09f" containerName="payload-processing"
Jul 24 17:26:07.224760 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.224693    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e546ab8a-6078-4a57-835f-12021c383f24" containerName="maas-api"
Jul 24 17:26:07.224760 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.224701    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e546ab8a-6078-4a57-835f-12021c383f24" containerName="maas-api"
Jul 24 17:26:07.224760 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.224713    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe1f3a66-3f28-418e-ae20-1d1bf066319e" containerName="payload-processing"
Jul 24 17:26:07.224760 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.224722    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe1f3a66-3f28-418e-ae20-1d1bf066319e" containerName="payload-processing"
Jul 24 17:26:07.224760 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.224742    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d85ecd6-1a1b-4609-be64-4c0d5a3f6526" containerName="payload-pre-processing"
Jul 24 17:26:07.224760 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.224751    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d85ecd6-1a1b-4609-be64-4c0d5a3f6526" containerName="payload-pre-processing"
Jul 24 17:26:07.225133 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.224771    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e84a08eb-ad2c-46ce-a1d5-721d80b9e130" containerName="maas-api"
Jul 24 17:26:07.225133 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.224780    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e84a08eb-ad2c-46ce-a1d5-721d80b9e130" containerName="maas-api"
Jul 24 17:26:07.225133 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.224789    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a7b6b92-9dd4-4465-81c0-c0904662bf43" containerName="payload-pre-processing"
Jul 24 17:26:07.225133 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.224797    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a7b6b92-9dd4-4465-81c0-c0904662bf43" containerName="payload-pre-processing"
Jul 24 17:26:07.225133 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.224883    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a7b6b92-9dd4-4465-81c0-c0904662bf43" containerName="payload-pre-processing"
Jul 24 17:26:07.225133 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.224895    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d85ecd6-1a1b-4609-be64-4c0d5a3f6526" containerName="payload-pre-processing"
Jul 24 17:26:07.225133 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.224907    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="103e7629-5bff-4486-be12-8ea3fc7bf09f" containerName="payload-processing"
Jul 24 17:26:07.225133 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.224917    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e546ab8a-6078-4a57-835f-12021c383f24" containerName="maas-api"
Jul 24 17:26:07.225133 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.224923    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e84a08eb-ad2c-46ce-a1d5-721d80b9e130" containerName="maas-api"
Jul 24 17:26:07.225133 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.224949    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe1f3a66-3f28-418e-ae20-1d1bf066319e" containerName="payload-processing"
Jul 24 17:26:07.227849 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.227829    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-de68c7-79ccd89d4f-sxqrz"
Jul 24 17:26:07.230791 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.230773    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-de68c7-serving-cert\""
Jul 24 17:26:07.243248 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.243224    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-de68c7-79ccd89d4f-sxqrz"]
Jul 24 17:26:07.256087 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.256059    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:26:07.291436 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.291373    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01094640-b811-412f-b9bb-2d3cc7900efa-openshift-service-ca\") pod \"maas-api-e2e-shared-a-de68c7-79ccd89d4f-sxqrz\" (UID: \"01094640-b811-412f-b9bb-2d3cc7900efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-de68c7-79ccd89d4f-sxqrz"
Jul 24 17:26:07.291436 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.291405    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01094640-b811-412f-b9bb-2d3cc7900efa-maas-api-tls\") pod \"maas-api-e2e-shared-a-de68c7-79ccd89d4f-sxqrz\" (UID: \"01094640-b811-412f-b9bb-2d3cc7900efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-de68c7-79ccd89d4f-sxqrz"
Jul 24 17:26:07.291577 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.291445    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nr68\" (UniqueName: \"kubernetes.io/projected/01094640-b811-412f-b9bb-2d3cc7900efa-kube-api-access-5nr68\") pod \"maas-api-e2e-shared-a-de68c7-79ccd89d4f-sxqrz\" (UID: \"01094640-b811-412f-b9bb-2d3cc7900efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-de68c7-79ccd89d4f-sxqrz"
Jul 24 17:26:07.392879 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.392841    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nr68\" (UniqueName: \"kubernetes.io/projected/01094640-b811-412f-b9bb-2d3cc7900efa-kube-api-access-5nr68\") pod \"maas-api-e2e-shared-a-de68c7-79ccd89d4f-sxqrz\" (UID: \"01094640-b811-412f-b9bb-2d3cc7900efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-de68c7-79ccd89d4f-sxqrz"
Jul 24 17:26:07.393121 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.392967    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01094640-b811-412f-b9bb-2d3cc7900efa-openshift-service-ca\") pod \"maas-api-e2e-shared-a-de68c7-79ccd89d4f-sxqrz\" (UID: \"01094640-b811-412f-b9bb-2d3cc7900efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-de68c7-79ccd89d4f-sxqrz"
Jul 24 17:26:07.393121 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.393000    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01094640-b811-412f-b9bb-2d3cc7900efa-maas-api-tls\") pod \"maas-api-e2e-shared-a-de68c7-79ccd89d4f-sxqrz\" (UID: \"01094640-b811-412f-b9bb-2d3cc7900efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-de68c7-79ccd89d4f-sxqrz"
Jul 24 17:26:07.393121 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:26:07.393113    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-de68c7-serving-cert: secret "maas-api-e2e-shared-a-de68c7-serving-cert" not found
Jul 24 17:26:07.393314 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:26:07.393172    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01094640-b811-412f-b9bb-2d3cc7900efa-maas-api-tls podName:01094640-b811-412f-b9bb-2d3cc7900efa nodeName:}" failed. No retries permitted until 2026-07-24 17:26:07.893155169 +0000 UTC m=+2590.535119033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/01094640-b811-412f-b9bb-2d3cc7900efa-maas-api-tls") pod "maas-api-e2e-shared-a-de68c7-79ccd89d4f-sxqrz" (UID: "01094640-b811-412f-b9bb-2d3cc7900efa") : secret "maas-api-e2e-shared-a-de68c7-serving-cert" not found
Jul 24 17:26:07.393688 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.393663    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01094640-b811-412f-b9bb-2d3cc7900efa-openshift-service-ca\") pod \"maas-api-e2e-shared-a-de68c7-79ccd89d4f-sxqrz\" (UID: \"01094640-b811-412f-b9bb-2d3cc7900efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-de68c7-79ccd89d4f-sxqrz"
Jul 24 17:26:07.413233 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.413202    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nr68\" (UniqueName: \"kubernetes.io/projected/01094640-b811-412f-b9bb-2d3cc7900efa-kube-api-access-5nr68\") pod \"maas-api-e2e-shared-a-de68c7-79ccd89d4f-sxqrz\" (UID: \"01094640-b811-412f-b9bb-2d3cc7900efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-de68c7-79ccd89d4f-sxqrz"
Jul 24 17:26:07.501111 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.501067    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:26:07.825901 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.825864    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:26:07.897979 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.897947    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01094640-b811-412f-b9bb-2d3cc7900efa-maas-api-tls\") pod \"maas-api-e2e-shared-a-de68c7-79ccd89d4f-sxqrz\" (UID: \"01094640-b811-412f-b9bb-2d3cc7900efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-de68c7-79ccd89d4f-sxqrz"
Jul 24 17:26:07.900422 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.900400    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01094640-b811-412f-b9bb-2d3cc7900efa-maas-api-tls\") pod \"maas-api-e2e-shared-a-de68c7-79ccd89d4f-sxqrz\" (UID: \"01094640-b811-412f-b9bb-2d3cc7900efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-de68c7-79ccd89d4f-sxqrz"
Jul 24 17:26:07.924589 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.924563    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-de68c7-7b7c5fd57b-dj4g8"]
Jul 24 17:26:07.925565 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.925540    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:26:07.928080 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.928062    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-de68c7-7b7c5fd57b-dj4g8"
Jul 24 17:26:07.930673 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.930655    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-de68c7\""
Jul 24 17:26:07.940467 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.940446    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-de68c7-7b7c5fd57b-dj4g8"]
Jul 24 17:26:07.962639 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.962609    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-de68c7-676cb7d874-q76wx"]
Jul 24 17:26:07.966503 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.966483    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-de68c7-676cb7d874-q76wx"
Jul 24 17:26:07.969823 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.969805    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-de68c7-dockercfg-qmz22\""
Jul 24 17:26:07.983358 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.983337    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-de68c7-676cb7d874-q76wx"]
Jul 24 17:26:07.998815 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.998793    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/402d7ca3-45e0-46c7-a954-050a4f86a6ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-de68c7-7b7c5fd57b-dj4g8\" (UID: \"402d7ca3-45e0-46c7-a954-050a4f86a6ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-de68c7-7b7c5fd57b-dj4g8"
Jul 24 17:26:07.998921 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:07.998829    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5qs4\" (UniqueName: \"kubernetes.io/projected/402d7ca3-45e0-46c7-a954-050a4f86a6ed-kube-api-access-r5qs4\") pod \"payload-pre-processing-e2e-shared-a-de68c7-7b7c5fd57b-dj4g8\" (UID: \"402d7ca3-45e0-46c7-a954-050a4f86a6ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-de68c7-7b7c5fd57b-dj4g8"
Jul 24 17:26:08.100069 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:08.099974    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwxc2\" (UniqueName: \"kubernetes.io/projected/deb9c86a-2b44-474f-8edd-a7e0183e137b-kube-api-access-nwxc2\") pod \"payload-processing-e2e-shared-a-de68c7-676cb7d874-q76wx\" (UID: \"deb9c86a-2b44-474f-8edd-a7e0183e137b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-de68c7-676cb7d874-q76wx"
Jul 24 17:26:08.100515 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:08.100099    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/402d7ca3-45e0-46c7-a954-050a4f86a6ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-de68c7-7b7c5fd57b-dj4g8\" (UID: \"402d7ca3-45e0-46c7-a954-050a4f86a6ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-de68c7-7b7c5fd57b-dj4g8"
Jul 24 17:26:08.100515 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:08.100133    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/deb9c86a-2b44-474f-8edd-a7e0183e137b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-de68c7-676cb7d874-q76wx\" (UID: \"deb9c86a-2b44-474f-8edd-a7e0183e137b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-de68c7-676cb7d874-q76wx"
Jul 24 17:26:08.100515 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:08.100164    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5qs4\" (UniqueName: \"kubernetes.io/projected/402d7ca3-45e0-46c7-a954-050a4f86a6ed-kube-api-access-r5qs4\") pod \"payload-pre-processing-e2e-shared-a-de68c7-7b7c5fd57b-dj4g8\" (UID: \"402d7ca3-45e0-46c7-a954-050a4f86a6ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-de68c7-7b7c5fd57b-dj4g8"
Jul 24 17:26:08.100888 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:08.100864    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/402d7ca3-45e0-46c7-a954-050a4f86a6ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-de68c7-7b7c5fd57b-dj4g8\" (UID: \"402d7ca3-45e0-46c7-a954-050a4f86a6ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-de68c7-7b7c5fd57b-dj4g8"
Jul 24 17:26:08.109609 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:08.109586    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5qs4\" (UniqueName: \"kubernetes.io/projected/402d7ca3-45e0-46c7-a954-050a4f86a6ed-kube-api-access-r5qs4\") pod \"payload-pre-processing-e2e-shared-a-de68c7-7b7c5fd57b-dj4g8\" (UID: \"402d7ca3-45e0-46c7-a954-050a4f86a6ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-de68c7-7b7c5fd57b-dj4g8"
Jul 24 17:26:08.137317 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:08.137292    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-de68c7-79ccd89d4f-sxqrz"
Jul 24 17:26:08.201070 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:08.200923    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/deb9c86a-2b44-474f-8edd-a7e0183e137b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-de68c7-676cb7d874-q76wx\" (UID: \"deb9c86a-2b44-474f-8edd-a7e0183e137b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-de68c7-676cb7d874-q76wx"
Jul 24 17:26:08.201237 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:08.201186    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwxc2\" (UniqueName: \"kubernetes.io/projected/deb9c86a-2b44-474f-8edd-a7e0183e137b-kube-api-access-nwxc2\") pod \"payload-processing-e2e-shared-a-de68c7-676cb7d874-q76wx\" (UID: \"deb9c86a-2b44-474f-8edd-a7e0183e137b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-de68c7-676cb7d874-q76wx"
Jul 24 17:26:08.202551 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:08.202281    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/deb9c86a-2b44-474f-8edd-a7e0183e137b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-de68c7-676cb7d874-q76wx\" (UID: \"deb9c86a-2b44-474f-8edd-a7e0183e137b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-de68c7-676cb7d874-q76wx"
Jul 24 17:26:08.211306 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:08.211282    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwxc2\" (UniqueName: \"kubernetes.io/projected/deb9c86a-2b44-474f-8edd-a7e0183e137b-kube-api-access-nwxc2\") pod \"payload-processing-e2e-shared-a-de68c7-676cb7d874-q76wx\" (UID: \"deb9c86a-2b44-474f-8edd-a7e0183e137b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-de68c7-676cb7d874-q76wx"
Jul 24 17:26:08.239281 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:08.239202    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-de68c7-7b7c5fd57b-dj4g8"
Jul 24 17:26:08.275706 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:08.275620    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-de68c7-79ccd89d4f-sxqrz"]
Jul 24 17:26:08.277483 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:26:08.277459    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01094640_b811_412f_b9bb_2d3cc7900efa.slice/crio-7c01c49358557cd16442e7379c3c770b67feca3e353cba254ef7df924c142526 WatchSource:0}: Error finding container 7c01c49358557cd16442e7379c3c770b67feca3e353cba254ef7df924c142526: Status 404 returned error can't find the container with id 7c01c49358557cd16442e7379c3c770b67feca3e353cba254ef7df924c142526
Jul 24 17:26:08.278654 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:08.278630    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-de68c7-676cb7d874-q76wx"
Jul 24 17:26:08.404307 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:08.404281    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-de68c7-7b7c5fd57b-dj4g8"]
Jul 24 17:26:08.405906 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:26:08.405857    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod402d7ca3_45e0_46c7_a954_050a4f86a6ed.slice/crio-36e46209db3c9d9f34062ddd40790b6a4a1220b5527b71cd7dad35c28fa0dadb WatchSource:0}: Error finding container 36e46209db3c9d9f34062ddd40790b6a4a1220b5527b71cd7dad35c28fa0dadb: Status 404 returned error can't find the container with id 36e46209db3c9d9f34062ddd40790b6a4a1220b5527b71cd7dad35c28fa0dadb
Jul 24 17:26:08.451213 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:08.451192    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-de68c7-676cb7d874-q76wx"]
Jul 24 17:26:08.459186 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:26:08.459162    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeb9c86a_2b44_474f_8edd_a7e0183e137b.slice/crio-3acae923a5f98b2de183e7f2527c37b885f7beb68610c3befda18c6d79c4c61f WatchSource:0}: Error finding container 3acae923a5f98b2de183e7f2527c37b885f7beb68610c3befda18c6d79c4c61f: Status 404 returned error can't find the container with id 3acae923a5f98b2de183e7f2527c37b885f7beb68610c3befda18c6d79c4c61f
Jul 24 17:26:08.965768 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:08.965664    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-de68c7-676cb7d874-q76wx" event={"ID":"deb9c86a-2b44-474f-8edd-a7e0183e137b","Type":"ContainerStarted","Data":"7788c7de6b943929be64e3c9f1b8b7f62a1989506050cc8767333be4fff5d301"}
Jul 24 17:26:08.965768 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:08.965733    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-de68c7-676cb7d874-q76wx" event={"ID":"deb9c86a-2b44-474f-8edd-a7e0183e137b","Type":"ContainerStarted","Data":"3acae923a5f98b2de183e7f2527c37b885f7beb68610c3befda18c6d79c4c61f"}
Jul 24 17:26:08.966031 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:08.965845    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-de68c7-676cb7d874-q76wx"
Jul 24 17:26:08.967228 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:08.967198    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-de68c7-7b7c5fd57b-dj4g8" event={"ID":"402d7ca3-45e0-46c7-a954-050a4f86a6ed","Type":"ContainerStarted","Data":"3394a5c8058c6a0d6a205c59aba3fb4cfb94e90748a7eb66104d5100597d4bd9"}
Jul 24 17:26:08.967376 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:08.967239    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-de68c7-7b7c5fd57b-dj4g8" event={"ID":"402d7ca3-45e0-46c7-a954-050a4f86a6ed","Type":"ContainerStarted","Data":"36e46209db3c9d9f34062ddd40790b6a4a1220b5527b71cd7dad35c28fa0dadb"}
Jul 24 17:26:08.967376 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:08.967265    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-de68c7-7b7c5fd57b-dj4g8"
Jul 24 17:26:08.968741 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:08.968718    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-de68c7-79ccd89d4f-sxqrz" event={"ID":"01094640-b811-412f-b9bb-2d3cc7900efa","Type":"ContainerStarted","Data":"93e31cd51ab19ee6bc05b590cd6643d28811ffbd4e0b20ad0122622f0fd9a7b8"}
Jul 24 17:26:08.968839 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:08.968759    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-de68c7-79ccd89d4f-sxqrz" event={"ID":"01094640-b811-412f-b9bb-2d3cc7900efa","Type":"ContainerStarted","Data":"7c01c49358557cd16442e7379c3c770b67feca3e353cba254ef7df924c142526"}
Jul 24 17:26:08.968839 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:08.968814    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-de68c7-79ccd89d4f-sxqrz"
Jul 24 17:26:08.984727 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:08.984680    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-de68c7-676cb7d874-q76wx" podStartSLOduration=1.984668318 podStartE2EDuration="1.984668318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:26:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:26:08.982424151 +0000 UTC m=+2591.624388032" watchObservedRunningTime="2026-07-24 17:26:08.984668318 +0000 UTC m=+2591.626632466"
Jul 24 17:26:09.001091 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:09.001055    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-de68c7-79ccd89d4f-sxqrz" podStartSLOduration=1.601797517 podStartE2EDuration="2.001043627s" totalImagesPullingTime="399.24611ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:26:08.998817535 +0000 UTC m=+2591.640781419" watchObservedRunningTime="2026-07-24 17:26:09.001043627 +0000 UTC m=+2591.643007521"
Jul 24 17:26:09.014212 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:09.014177    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-de68c7-7b7c5fd57b-dj4g8" podStartSLOduration=2.014165097 podStartE2EDuration="2.014165097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:26:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:26:09.013638085 +0000 UTC m=+2591.655602014" watchObservedRunningTime="2026-07-24 17:26:09.014165097 +0000 UTC m=+2591.656128980"
Jul 24 17:26:11.437007 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:11.436956    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:26:11.757438 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:11.757327    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:26:11.819698 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:11.819669    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:26:12.107652 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:12.107544    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:26:12.394179 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:12.394148    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:26:14.981906 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:14.981876    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-de68c7-79ccd89d4f-sxqrz"
Jul 24 17:26:15.082770 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:15.082738    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:26:15.206863 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:15.206826    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d73e95-7f4cdbc78-5x8f9"]
Jul 24 17:26:15.210577 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:15.210555    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d73e95-7f4cdbc78-5x8f9"
Jul 24 17:26:15.215298 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:15.215281    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-d73e95-serving-cert\""
Jul 24 17:26:15.221343 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:15.221321    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:26:15.226063 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:15.226044    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d73e95-7f4cdbc78-5x8f9"]
Jul 24 17:26:15.268430 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:15.268363    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19014f4d-b299-4de5-93bd-a1c030ad60c7-maas-api-tls\") pod \"maas-api-e2e-shared-b-d73e95-7f4cdbc78-5x8f9\" (UID: \"19014f4d-b299-4de5-93bd-a1c030ad60c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d73e95-7f4cdbc78-5x8f9"
Jul 24 17:26:15.268430 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:15.268419    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7v8h\" (UniqueName: \"kubernetes.io/projected/19014f4d-b299-4de5-93bd-a1c030ad60c7-kube-api-access-h7v8h\") pod \"maas-api-e2e-shared-b-d73e95-7f4cdbc78-5x8f9\" (UID: \"19014f4d-b299-4de5-93bd-a1c030ad60c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d73e95-7f4cdbc78-5x8f9"
Jul 24 17:26:15.268585 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:15.268486    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19014f4d-b299-4de5-93bd-a1c030ad60c7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-d73e95-7f4cdbc78-5x8f9\" (UID: \"19014f4d-b299-4de5-93bd-a1c030ad60c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d73e95-7f4cdbc78-5x8f9"
Jul 24 17:26:15.358012 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:15.357979    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:26:15.369179 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:15.369147    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19014f4d-b299-4de5-93bd-a1c030ad60c7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-d73e95-7f4cdbc78-5x8f9\" (UID: \"19014f4d-b299-4de5-93bd-a1c030ad60c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d73e95-7f4cdbc78-5x8f9"
Jul 24 17:26:15.369301 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:15.369232    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19014f4d-b299-4de5-93bd-a1c030ad60c7-maas-api-tls\") pod \"maas-api-e2e-shared-b-d73e95-7f4cdbc78-5x8f9\" (UID: \"19014f4d-b299-4de5-93bd-a1c030ad60c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d73e95-7f4cdbc78-5x8f9"
Jul 24 17:26:15.369301 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:15.369282    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7v8h\" (UniqueName: \"kubernetes.io/projected/19014f4d-b299-4de5-93bd-a1c030ad60c7-kube-api-access-h7v8h\") pod \"maas-api-e2e-shared-b-d73e95-7f4cdbc78-5x8f9\" (UID: \"19014f4d-b299-4de5-93bd-a1c030ad60c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d73e95-7f4cdbc78-5x8f9"
Jul 24 17:26:15.369395 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:26:15.369377    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-d73e95-serving-cert: secret "maas-api-e2e-shared-b-d73e95-serving-cert" not found
Jul 24 17:26:15.369460 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:26:15.369450    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19014f4d-b299-4de5-93bd-a1c030ad60c7-maas-api-tls podName:19014f4d-b299-4de5-93bd-a1c030ad60c7 nodeName:}" failed. No retries permitted until 2026-07-24 17:26:15.86943467 +0000 UTC m=+2598.511398531 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/19014f4d-b299-4de5-93bd-a1c030ad60c7-maas-api-tls") pod "maas-api-e2e-shared-b-d73e95-7f4cdbc78-5x8f9" (UID: "19014f4d-b299-4de5-93bd-a1c030ad60c7") : secret "maas-api-e2e-shared-b-d73e95-serving-cert" not found
Jul 24 17:26:15.369740 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:15.369723    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19014f4d-b299-4de5-93bd-a1c030ad60c7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-d73e95-7f4cdbc78-5x8f9\" (UID: \"19014f4d-b299-4de5-93bd-a1c030ad60c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d73e95-7f4cdbc78-5x8f9"
Jul 24 17:26:15.380905 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:15.380886    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7v8h\" (UniqueName: \"kubernetes.io/projected/19014f4d-b299-4de5-93bd-a1c030ad60c7-kube-api-access-h7v8h\") pod \"maas-api-e2e-shared-b-d73e95-7f4cdbc78-5x8f9\" (UID: \"19014f4d-b299-4de5-93bd-a1c030ad60c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d73e95-7f4cdbc78-5x8f9"
Jul 24 17:26:15.495877 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:15.495827    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:26:15.874295 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:15.874260    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19014f4d-b299-4de5-93bd-a1c030ad60c7-maas-api-tls\") pod \"maas-api-e2e-shared-b-d73e95-7f4cdbc78-5x8f9\" (UID: \"19014f4d-b299-4de5-93bd-a1c030ad60c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d73e95-7f4cdbc78-5x8f9"
Jul 24 17:26:15.876689 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:15.876669    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19014f4d-b299-4de5-93bd-a1c030ad60c7-maas-api-tls\") pod \"maas-api-e2e-shared-b-d73e95-7f4cdbc78-5x8f9\" (UID: \"19014f4d-b299-4de5-93bd-a1c030ad60c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d73e95-7f4cdbc78-5x8f9"
Jul 24 17:26:16.076278 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:16.076250    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:26:16.121562 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:16.121520    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d73e95-7f4cdbc78-5x8f9"
Jul 24 17:26:16.234981 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:16.234924    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:26:16.275085 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:16.274982    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d73e95-7f4cdbc78-5x8f9"]
Jul 24 17:26:16.277994 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:26:16.277960    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19014f4d_b299_4de5_93bd_a1c030ad60c7.slice/crio-2c4465889a8afdece48038299dc3fa72e1827dae7eb73f648d27035459f6630b WatchSource:0}: Error finding container 2c4465889a8afdece48038299dc3fa72e1827dae7eb73f648d27035459f6630b: Status 404 returned error can't find the container with id 2c4465889a8afdece48038299dc3fa72e1827dae7eb73f648d27035459f6630b
Jul 24 17:26:16.309279 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:16.309257    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-d73e95-579c4f64d-cpqn6"]
Jul 24 17:26:16.313920 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:16.313903    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d73e95-579c4f64d-cpqn6"
Jul 24 17:26:16.319299 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:16.319282    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-d73e95\""
Jul 24 17:26:16.327712 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:16.327689    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-d73e95-579c4f64d-cpqn6"]
Jul 24 17:26:16.356674 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:16.356647    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-d73e95-5b677fb66b-254qn"]
Jul 24 17:26:16.360049 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:16.360035    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-d73e95-5b677fb66b-254qn"
Jul 24 17:26:16.363467 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:16.363448    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-d73e95-dockercfg-4xtcn\""
Jul 24 17:26:16.378291 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:16.378241    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65bebe11-06f7-4960-ab05-3cdcfb8432c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-d73e95-579c4f64d-cpqn6\" (UID: \"65bebe11-06f7-4960-ab05-3cdcfb8432c2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d73e95-579c4f64d-cpqn6"
Jul 24 17:26:16.378291 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:16.378269    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hdgc\" (UniqueName: \"kubernetes.io/projected/65bebe11-06f7-4960-ab05-3cdcfb8432c2-kube-api-access-7hdgc\") pod \"payload-pre-processing-e2e-shared-b-d73e95-579c4f64d-cpqn6\" (UID: \"65bebe11-06f7-4960-ab05-3cdcfb8432c2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d73e95-579c4f64d-cpqn6"
Jul 24 17:26:16.388341 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:16.388319    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-d73e95-5b677fb66b-254qn"]
Jul 24 17:26:16.479825 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:16.479794    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65bebe11-06f7-4960-ab05-3cdcfb8432c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-d73e95-579c4f64d-cpqn6\" (UID: \"65bebe11-06f7-4960-ab05-3cdcfb8432c2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d73e95-579c4f64d-cpqn6"
Jul 24 17:26:16.480040 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:16.479836    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8rcp\" (UniqueName: \"kubernetes.io/projected/0c04556e-ad7a-431c-b4a9-a9372f613cfa-kube-api-access-n8rcp\") pod \"payload-processing-e2e-shared-b-d73e95-5b677fb66b-254qn\" (UID: \"0c04556e-ad7a-431c-b4a9-a9372f613cfa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-d73e95-5b677fb66b-254qn"
Jul 24 17:26:16.480040 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:16.479867    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hdgc\" (UniqueName: \"kubernetes.io/projected/65bebe11-06f7-4960-ab05-3cdcfb8432c2-kube-api-access-7hdgc\") pod \"payload-pre-processing-e2e-shared-b-d73e95-579c4f64d-cpqn6\" (UID: \"65bebe11-06f7-4960-ab05-3cdcfb8432c2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d73e95-579c4f64d-cpqn6"
Jul 24 17:26:16.480040 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:16.479916    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c04556e-ad7a-431c-b4a9-a9372f613cfa-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-d73e95-5b677fb66b-254qn\" (UID: \"0c04556e-ad7a-431c-b4a9-a9372f613cfa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-d73e95-5b677fb66b-254qn"
Jul 24 17:26:16.481030 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:16.481000    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65bebe11-06f7-4960-ab05-3cdcfb8432c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-d73e95-579c4f64d-cpqn6\" (UID: \"65bebe11-06f7-4960-ab05-3cdcfb8432c2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d73e95-579c4f64d-cpqn6"
Jul 24 17:26:16.493400 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:16.493376    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hdgc\" (UniqueName: \"kubernetes.io/projected/65bebe11-06f7-4960-ab05-3cdcfb8432c2-kube-api-access-7hdgc\") pod \"payload-pre-processing-e2e-shared-b-d73e95-579c4f64d-cpqn6\" (UID: \"65bebe11-06f7-4960-ab05-3cdcfb8432c2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d73e95-579c4f64d-cpqn6"
Jul 24 17:26:16.580669 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:16.580639    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8rcp\" (UniqueName: \"kubernetes.io/projected/0c04556e-ad7a-431c-b4a9-a9372f613cfa-kube-api-access-n8rcp\") pod \"payload-processing-e2e-shared-b-d73e95-5b677fb66b-254qn\" (UID: \"0c04556e-ad7a-431c-b4a9-a9372f613cfa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-d73e95-5b677fb66b-254qn"
Jul 24 17:26:16.580833 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:16.580700    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c04556e-ad7a-431c-b4a9-a9372f613cfa-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-d73e95-5b677fb66b-254qn\" (UID: \"0c04556e-ad7a-431c-b4a9-a9372f613cfa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-d73e95-5b677fb66b-254qn"
Jul 24 17:26:16.581546 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:16.581524    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c04556e-ad7a-431c-b4a9-a9372f613cfa-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-d73e95-5b677fb66b-254qn\" (UID: \"0c04556e-ad7a-431c-b4a9-a9372f613cfa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-d73e95-5b677fb66b-254qn"
Jul 24 17:26:16.592638 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:16.592615    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8rcp\" (UniqueName: \"kubernetes.io/projected/0c04556e-ad7a-431c-b4a9-a9372f613cfa-kube-api-access-n8rcp\") pod \"payload-processing-e2e-shared-b-d73e95-5b677fb66b-254qn\" (UID: \"0c04556e-ad7a-431c-b4a9-a9372f613cfa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-d73e95-5b677fb66b-254qn"
Jul 24 17:26:16.624366 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:16.624334    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d73e95-579c4f64d-cpqn6"
Jul 24 17:26:16.669720 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:16.669688    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-d73e95-5b677fb66b-254qn"
Jul 24 17:26:16.780211 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:16.780176    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-d73e95-579c4f64d-cpqn6"]
Jul 24 17:26:16.783898 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:26:16.783870    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65bebe11_06f7_4960_ab05_3cdcfb8432c2.slice/crio-6f5697f450e69332f4ebbb97eeae2eb7ac794d52a6e53d3a87f831530c762406 WatchSource:0}: Error finding container 6f5697f450e69332f4ebbb97eeae2eb7ac794d52a6e53d3a87f831530c762406: Status 404 returned error can't find the container with id 6f5697f450e69332f4ebbb97eeae2eb7ac794d52a6e53d3a87f831530c762406
Jul 24 17:26:16.847803 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:16.847776    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-d73e95-5b677fb66b-254qn"]
Jul 24 17:26:16.849097 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:26:16.849069    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c04556e_ad7a_431c_b4a9_a9372f613cfa.slice/crio-f770d951e08a3f688bde59409a14490a3edacad2c67d3e1da762e4f0eb88cdda WatchSource:0}: Error finding container f770d951e08a3f688bde59409a14490a3edacad2c67d3e1da762e4f0eb88cdda: Status 404 returned error can't find the container with id f770d951e08a3f688bde59409a14490a3edacad2c67d3e1da762e4f0eb88cdda
Jul 24 17:26:17.012347 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:17.012313    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d73e95-579c4f64d-cpqn6" event={"ID":"65bebe11-06f7-4960-ab05-3cdcfb8432c2","Type":"ContainerStarted","Data":"8feb4b13231552c839c84b0192bf5b6896475cad2a733236fa4640759bbb14e3"}
Jul 24 17:26:17.012534 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:17.012375    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d73e95-579c4f64d-cpqn6" event={"ID":"65bebe11-06f7-4960-ab05-3cdcfb8432c2","Type":"ContainerStarted","Data":"6f5697f450e69332f4ebbb97eeae2eb7ac794d52a6e53d3a87f831530c762406"}
Jul 24 17:26:17.012534 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:17.012403    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d73e95-579c4f64d-cpqn6"
Jul 24 17:26:17.014091 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:17.014065    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d73e95-7f4cdbc78-5x8f9" event={"ID":"19014f4d-b299-4de5-93bd-a1c030ad60c7","Type":"ContainerStarted","Data":"2ea25235b7a32c791c7bad00f425a87b8bf5ae3a213ea13fdd676cdf9fbcfde4"}
Jul 24 17:26:17.014224 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:17.014113    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d73e95-7f4cdbc78-5x8f9" event={"ID":"19014f4d-b299-4de5-93bd-a1c030ad60c7","Type":"ContainerStarted","Data":"2c4465889a8afdece48038299dc3fa72e1827dae7eb73f648d27035459f6630b"}
Jul 24 17:26:17.014224 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:17.014143    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d73e95-7f4cdbc78-5x8f9"
Jul 24 17:26:17.015660 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:17.015637    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-d73e95-5b677fb66b-254qn" event={"ID":"0c04556e-ad7a-431c-b4a9-a9372f613cfa","Type":"ContainerStarted","Data":"cc25f31bc043c0b455eb75699367e3bcacd8a6e50c1cdeddc78ef89a4232e4ef"}
Jul 24 17:26:17.015762 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:17.015675    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-d73e95-5b677fb66b-254qn" event={"ID":"0c04556e-ad7a-431c-b4a9-a9372f613cfa","Type":"ContainerStarted","Data":"f770d951e08a3f688bde59409a14490a3edacad2c67d3e1da762e4f0eb88cdda"}
Jul 24 17:26:17.015823 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:17.015762    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-d73e95-5b677fb66b-254qn"
Jul 24 17:26:17.030915 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:17.030875    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d73e95-579c4f64d-cpqn6" podStartSLOduration=1.030862194 podStartE2EDuration="1.030862194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:26:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:26:17.028606576 +0000 UTC m=+2599.670570484" watchObservedRunningTime="2026-07-24 17:26:17.030862194 +0000 UTC m=+2599.672826077"
Jul 24 17:26:17.055548 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:17.055511    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-d73e95-5b677fb66b-254qn" podStartSLOduration=1.055496952 podStartE2EDuration="1.055496952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:26:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:26:17.053623425 +0000 UTC m=+2599.695587309" watchObservedRunningTime="2026-07-24 17:26:17.055496952 +0000 UTC m=+2599.697460837"
Jul 24 17:26:17.090879 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:17.090824    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d73e95-7f4cdbc78-5x8f9" podStartSLOduration=1.73918153 podStartE2EDuration="2.090805875s" totalImagesPullingTime="351.624345ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:26:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:26:17.085107779 +0000 UTC m=+2599.727071685" watchObservedRunningTime="2026-07-24 17:26:17.090805875 +0000 UTC m=+2599.732769761"
Jul 24 17:26:19.642398 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:19.642353    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:26:19.977865 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:19.977793    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-de68c7-7b7c5fd57b-dj4g8"
Jul 24 17:26:19.978015 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:19.977872    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-de68c7-676cb7d874-q76wx"
Jul 24 17:26:23.027328 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:23.027299    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d73e95-7f4cdbc78-5x8f9"
Jul 24 17:26:28.022195 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:28.022146    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d73e95-579c4f64d-cpqn6" podUID="65bebe11-06f7-4960-ab05-3cdcfb8432c2" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.78:9004: connect: connection refused"
Jul 24 17:26:28.022721 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:28.022571    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-d73e95-5b677fb66b-254qn"
Jul 24 17:26:38.021706 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:38.021671    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d73e95-579c4f64d-cpqn6"
Jul 24 17:26:40.308710 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:40.308674    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:26:55.610813 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:26:55.610784    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:27:07.273271 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:27:07.273239    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:27:58.174186 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:27:58.174153    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ddklx_fb5a9838-1e05-4f86-b46c-760bd7aaa038/console-operator/1.log"
Jul 24 17:27:58.184368 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:27:58.184349    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ddklx_fb5a9838-1e05-4f86-b46c-760bd7aaa038/console-operator/1.log"
Jul 24 17:29:03.993781 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:29:03.993747    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:29:04.892570 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:29:04.892541    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:29:30.213033 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:29:30.212929    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:30:14.400157 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:30:14.400126    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:30:30.996828 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:30:30.996794    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:30:59.398319 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:30:59.398287    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:31:00.241371 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:31:00.241343    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:31:11.314419 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:31:11.314367    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:32:01.030089 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:32:01.030058    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:32:13.505524 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:32:13.505490    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:32:58.204622 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:32:58.204596    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ddklx_fb5a9838-1e05-4f86-b46c-760bd7aaa038/console-operator/1.log"
Jul 24 17:32:58.215772 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:32:58.215750    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ddklx_fb5a9838-1e05-4f86-b46c-760bd7aaa038/console-operator/1.log"
Jul 24 17:32:59.073198 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:32:59.073155    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:00.054263 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:00.054229    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:19.570450 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:19.570395    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:19.958860 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:19.958817    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:20.308855 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:20.308770    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:22.586555 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:22.586515    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:22.778411 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:22.778359    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:22.890843 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:22.890808    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:23.068273 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:23.068246    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:23.537919 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:23.537887    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:23.625796 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:23.625765    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:23.728193 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:23.728157    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6a1c15-6d5f5b75bd-crz8r"]
Jul 24 17:33:23.731822 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:23.731805    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a1c15-6d5f5b75bd-crz8r"
Jul 24 17:33:23.735389 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:23.735369    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6a1c15\""
Jul 24 17:33:23.735502 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:23.735393    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-6a1c15-dockercfg-4ntq2\""
Jul 24 17:33:23.744449 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:23.744426    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6a1c15-6d5f5b75bd-crz8r"]
Jul 24 17:33:23.753712 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:23.753681    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69a0d029-1031-4849-9060-ff5fb0e35065-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6a1c15-6d5f5b75bd-crz8r\" (UID: \"69a0d029-1031-4849-9060-ff5fb0e35065\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a1c15-6d5f5b75bd-crz8r"
Jul 24 17:33:23.753839 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:23.753717    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shtpb\" (UniqueName: \"kubernetes.io/projected/69a0d029-1031-4849-9060-ff5fb0e35065-kube-api-access-shtpb\") pod \"payload-pre-processing-e2e-ipp-a-6a1c15-6d5f5b75bd-crz8r\" (UID: \"69a0d029-1031-4849-9060-ff5fb0e35065\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a1c15-6d5f5b75bd-crz8r"
Jul 24 17:33:23.775149 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:23.775117    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6a1c15-c6d459c76-zp68x"]
Jul 24 17:33:23.779416 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:23.779395    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a1c15-c6d459c76-zp68x"
Jul 24 17:33:23.790639 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:23.790582    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6a1c15-c6d459c76-zp68x"]
Jul 24 17:33:23.854339 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:23.854306    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69a0d029-1031-4849-9060-ff5fb0e35065-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6a1c15-6d5f5b75bd-crz8r\" (UID: \"69a0d029-1031-4849-9060-ff5fb0e35065\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a1c15-6d5f5b75bd-crz8r"
Jul 24 17:33:23.854339 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:23.854340    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shtpb\" (UniqueName: \"kubernetes.io/projected/69a0d029-1031-4849-9060-ff5fb0e35065-kube-api-access-shtpb\") pod \"payload-pre-processing-e2e-ipp-a-6a1c15-6d5f5b75bd-crz8r\" (UID: \"69a0d029-1031-4849-9060-ff5fb0e35065\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a1c15-6d5f5b75bd-crz8r"
Jul 24 17:33:23.854545 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:23.854408    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01d35db8-e624-4941-9212-78752dc2c387-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6a1c15-c6d459c76-zp68x\" (UID: \"01d35db8-e624-4941-9212-78752dc2c387\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6a1c15-c6d459c76-zp68x"
Jul 24 17:33:23.854545 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:23.854429    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcq54\" (UniqueName: \"kubernetes.io/projected/01d35db8-e624-4941-9212-78752dc2c387-kube-api-access-lcq54\") pod \"payload-processing-e2e-ipp-a-6a1c15-c6d459c76-zp68x\" (UID: \"01d35db8-e624-4941-9212-78752dc2c387\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6a1c15-c6d459c76-zp68x"
Jul 24 17:33:23.855025 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:23.855005    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69a0d029-1031-4849-9060-ff5fb0e35065-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6a1c15-6d5f5b75bd-crz8r\" (UID: \"69a0d029-1031-4849-9060-ff5fb0e35065\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a1c15-6d5f5b75bd-crz8r"
Jul 24 17:33:23.863442 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:23.863422    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shtpb\" (UniqueName: \"kubernetes.io/projected/69a0d029-1031-4849-9060-ff5fb0e35065-kube-api-access-shtpb\") pod \"payload-pre-processing-e2e-ipp-a-6a1c15-6d5f5b75bd-crz8r\" (UID: \"69a0d029-1031-4849-9060-ff5fb0e35065\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a1c15-6d5f5b75bd-crz8r"
Jul 24 17:33:23.955774 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:23.955741    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01d35db8-e624-4941-9212-78752dc2c387-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6a1c15-c6d459c76-zp68x\" (UID: \"01d35db8-e624-4941-9212-78752dc2c387\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6a1c15-c6d459c76-zp68x"
Jul 24 17:33:23.955774 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:23.955777    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcq54\" (UniqueName: \"kubernetes.io/projected/01d35db8-e624-4941-9212-78752dc2c387-kube-api-access-lcq54\") pod \"payload-processing-e2e-ipp-a-6a1c15-c6d459c76-zp68x\" (UID: \"01d35db8-e624-4941-9212-78752dc2c387\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6a1c15-c6d459c76-zp68x"
Jul 24 17:33:23.956511 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:23.956492    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01d35db8-e624-4941-9212-78752dc2c387-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6a1c15-c6d459c76-zp68x\" (UID: \"01d35db8-e624-4941-9212-78752dc2c387\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6a1c15-c6d459c76-zp68x"
Jul 24 17:33:23.965118 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:23.965083    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcq54\" (UniqueName: \"kubernetes.io/projected/01d35db8-e624-4941-9212-78752dc2c387-kube-api-access-lcq54\") pod \"payload-processing-e2e-ipp-a-6a1c15-c6d459c76-zp68x\" (UID: \"01d35db8-e624-4941-9212-78752dc2c387\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6a1c15-c6d459c76-zp68x"
Jul 24 17:33:24.043193 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:24.043124    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a1c15-6d5f5b75bd-crz8r"
Jul 24 17:33:24.091862 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:24.091830    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a1c15-c6d459c76-zp68x"
Jul 24 17:33:24.245801 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:24.245773    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6a1c15-c6d459c76-zp68x"]
Jul 24 17:33:24.247192 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:33:24.247165    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01d35db8_e624_4941_9212_78752dc2c387.slice/crio-aa9a15f0b6ddfdd559720b87a7a892d38cede6098c8835f91424190565772c55 WatchSource:0}: Error finding container aa9a15f0b6ddfdd559720b87a7a892d38cede6098c8835f91424190565772c55: Status 404 returned error can't find the container with id aa9a15f0b6ddfdd559720b87a7a892d38cede6098c8835f91424190565772c55
Jul 24 17:33:24.400247 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:24.400185    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6a1c15-6d5f5b75bd-crz8r"]
Jul 24 17:33:24.402503 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:33:24.402478    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a0d029_1031_4849_9060_ff5fb0e35065.slice/crio-9189aefb3121a0a52e4caf3f7585abfb7bf461766d664bff0465ca29393c8126 WatchSource:0}: Error finding container 9189aefb3121a0a52e4caf3f7585abfb7bf461766d664bff0465ca29393c8126: Status 404 returned error can't find the container with id 9189aefb3121a0a52e4caf3f7585abfb7bf461766d664bff0465ca29393c8126
Jul 24 17:33:25.168124 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:25.168090    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a1c15-6d5f5b75bd-crz8r" event={"ID":"69a0d029-1031-4849-9060-ff5fb0e35065","Type":"ContainerStarted","Data":"97cd71a094bb3f40cfe3f85c6dba6dd40bd07add6f1117d6a948629ed831abbc"}
Jul 24 17:33:25.168575 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:25.168142    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a1c15-6d5f5b75bd-crz8r" event={"ID":"69a0d029-1031-4849-9060-ff5fb0e35065","Type":"ContainerStarted","Data":"9189aefb3121a0a52e4caf3f7585abfb7bf461766d664bff0465ca29393c8126"}
Jul 24 17:33:25.168575 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:25.168166    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a1c15-6d5f5b75bd-crz8r"
Jul 24 17:33:25.169657 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:25.169626    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a1c15-c6d459c76-zp68x" event={"ID":"01d35db8-e624-4941-9212-78752dc2c387","Type":"ContainerStarted","Data":"d84c2af317e2f3bac6c03527d2090036c27af17ff0950a3d69cc1572155c4b05"}
Jul 24 17:33:25.169787 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:25.169682    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a1c15-c6d459c76-zp68x" event={"ID":"01d35db8-e624-4941-9212-78752dc2c387","Type":"ContainerStarted","Data":"aa9a15f0b6ddfdd559720b87a7a892d38cede6098c8835f91424190565772c55"}
Jul 24 17:33:25.169787 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:25.169713    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a1c15-c6d459c76-zp68x"
Jul 24 17:33:25.209037 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:25.208972    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a1c15-c6d459c76-zp68x" podStartSLOduration=2.208956523 podStartE2EDuration="2.208956523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:33:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:33:25.207096746 +0000 UTC m=+3027.849060649" watchObservedRunningTime="2026-07-24 17:33:25.208956523 +0000 UTC m=+3027.850920401"
Jul 24 17:33:25.209262 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:25.209242    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a1c15-6d5f5b75bd-crz8r" podStartSLOduration=2.20923553 podStartE2EDuration="2.20923553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:33:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:33:25.187395338 +0000 UTC m=+3027.829359222" watchObservedRunningTime="2026-07-24 17:33:25.20923553 +0000 UTC m=+3027.851199474"
Jul 24 17:33:27.045794 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:27.045759    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:36.177285 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:36.177252    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a1c15-6d5f5b75bd-crz8r"
Jul 24 17:33:36.177656 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:36.177410    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a1c15-c6d459c76-zp68x"
Jul 24 17:33:38.028676 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:38.028642    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:38.435167 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:38.435137    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:38.964243 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:38.964210    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:41.702224 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:41.702185    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:41.801526 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:41.801486    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:42.005018 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:42.004920    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:42.148386 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:42.148342    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:42.691532 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:42.691494    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:42.816166 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:42.816132    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:42.849817 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:42.849779    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a2f844-9b679bb8-gzfxk"]
Jul 24 17:33:42.853536 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:42.853512    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2f844-9b679bb8-gzfxk"
Jul 24 17:33:42.857401 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:42.857377    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a2f844\""
Jul 24 17:33:42.857625 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:42.857608    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a2f844-dockercfg-bg8pb\""
Jul 24 17:33:42.873476 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:42.873442    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a2f844-9b679bb8-gzfxk"]
Jul 24 17:33:42.931367 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:42.931332    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9fecf19-4c51-4a7c-8036-e64822d7ba44-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a2f844-9b679bb8-gzfxk\" (UID: \"f9fecf19-4c51-4a7c-8036-e64822d7ba44\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2f844-9b679bb8-gzfxk"
Jul 24 17:33:42.931548 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:42.931376    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzcpg\" (UniqueName: \"kubernetes.io/projected/f9fecf19-4c51-4a7c-8036-e64822d7ba44-kube-api-access-dzcpg\") pod \"payload-pre-processing-e2e-ipp-b-a2f844-9b679bb8-gzfxk\" (UID: \"f9fecf19-4c51-4a7c-8036-e64822d7ba44\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2f844-9b679bb8-gzfxk"
Jul 24 17:33:42.956274 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:42.956197    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a2f844-8845757cd-289sl"]
Jul 24 17:33:42.960082 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:42.960060    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2f844-8845757cd-289sl"
Jul 24 17:33:42.972097 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:42.972070    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a2f844-8845757cd-289sl"]
Jul 24 17:33:43.032255 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:43.032217    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f288f4c-fda7-45ea-b4a7-dfb5f463816a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a2f844-8845757cd-289sl\" (UID: \"5f288f4c-fda7-45ea-b4a7-dfb5f463816a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a2f844-8845757cd-289sl"
Jul 24 17:33:43.032422 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:43.032266    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc482\" (UniqueName: \"kubernetes.io/projected/5f288f4c-fda7-45ea-b4a7-dfb5f463816a-kube-api-access-mc482\") pod \"payload-processing-e2e-ipp-b-a2f844-8845757cd-289sl\" (UID: \"5f288f4c-fda7-45ea-b4a7-dfb5f463816a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a2f844-8845757cd-289sl"
Jul 24 17:33:43.032422 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:43.032333    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9fecf19-4c51-4a7c-8036-e64822d7ba44-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a2f844-9b679bb8-gzfxk\" (UID: \"f9fecf19-4c51-4a7c-8036-e64822d7ba44\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2f844-9b679bb8-gzfxk"
Jul 24 17:33:43.032422 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:43.032363    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzcpg\" (UniqueName: \"kubernetes.io/projected/f9fecf19-4c51-4a7c-8036-e64822d7ba44-kube-api-access-dzcpg\") pod \"payload-pre-processing-e2e-ipp-b-a2f844-9b679bb8-gzfxk\" (UID: \"f9fecf19-4c51-4a7c-8036-e64822d7ba44\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2f844-9b679bb8-gzfxk"
Jul 24 17:33:43.033010 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:43.032992    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9fecf19-4c51-4a7c-8036-e64822d7ba44-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a2f844-9b679bb8-gzfxk\" (UID: \"f9fecf19-4c51-4a7c-8036-e64822d7ba44\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2f844-9b679bb8-gzfxk"
Jul 24 17:33:43.041632 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:43.041605    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzcpg\" (UniqueName: \"kubernetes.io/projected/f9fecf19-4c51-4a7c-8036-e64822d7ba44-kube-api-access-dzcpg\") pod \"payload-pre-processing-e2e-ipp-b-a2f844-9b679bb8-gzfxk\" (UID: \"f9fecf19-4c51-4a7c-8036-e64822d7ba44\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2f844-9b679bb8-gzfxk"
Jul 24 17:33:43.133174 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:43.133132    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc482\" (UniqueName: \"kubernetes.io/projected/5f288f4c-fda7-45ea-b4a7-dfb5f463816a-kube-api-access-mc482\") pod \"payload-processing-e2e-ipp-b-a2f844-8845757cd-289sl\" (UID: \"5f288f4c-fda7-45ea-b4a7-dfb5f463816a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a2f844-8845757cd-289sl"
Jul 24 17:33:43.133383 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:43.133290    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f288f4c-fda7-45ea-b4a7-dfb5f463816a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a2f844-8845757cd-289sl\" (UID: \"5f288f4c-fda7-45ea-b4a7-dfb5f463816a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a2f844-8845757cd-289sl"
Jul 24 17:33:43.134092 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:43.134065    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f288f4c-fda7-45ea-b4a7-dfb5f463816a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a2f844-8845757cd-289sl\" (UID: \"5f288f4c-fda7-45ea-b4a7-dfb5f463816a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a2f844-8845757cd-289sl"
Jul 24 17:33:43.143759 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:43.143732    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc482\" (UniqueName: \"kubernetes.io/projected/5f288f4c-fda7-45ea-b4a7-dfb5f463816a-kube-api-access-mc482\") pod \"payload-processing-e2e-ipp-b-a2f844-8845757cd-289sl\" (UID: \"5f288f4c-fda7-45ea-b4a7-dfb5f463816a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a2f844-8845757cd-289sl"
Jul 24 17:33:43.165126 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:43.165096    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2f844-9b679bb8-gzfxk"
Jul 24 17:33:43.271157 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:43.271124    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2f844-8845757cd-289sl"
Jul 24 17:33:43.318956 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:43.318897    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a2f844-9b679bb8-gzfxk"]
Jul 24 17:33:43.321097 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:33:43.320999    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9fecf19_4c51_4a7c_8036_e64822d7ba44.slice/crio-fb1ddd762bc68146a4e231773164e3f3c781d7602e127a55146c9a0e96879330 WatchSource:0}: Error finding container fb1ddd762bc68146a4e231773164e3f3c781d7602e127a55146c9a0e96879330: Status 404 returned error can't find the container with id fb1ddd762bc68146a4e231773164e3f3c781d7602e127a55146c9a0e96879330
Jul 24 17:33:43.435416 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:43.435355    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a2f844-8845757cd-289sl"]
Jul 24 17:33:43.437810 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:33:43.437785    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f288f4c_fda7_45ea_b4a7_dfb5f463816a.slice/crio-56505f0a8ba3b8d13caf14ffaf257622a26b9941368b95ab7b2c6d348fbf874b WatchSource:0}: Error finding container 56505f0a8ba3b8d13caf14ffaf257622a26b9941368b95ab7b2c6d348fbf874b: Status 404 returned error can't find the container with id 56505f0a8ba3b8d13caf14ffaf257622a26b9941368b95ab7b2c6d348fbf874b
Jul 24 17:33:44.271311 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:44.271270    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2f844-9b679bb8-gzfxk" event={"ID":"f9fecf19-4c51-4a7c-8036-e64822d7ba44","Type":"ContainerStarted","Data":"3bc0810e92eb7c26d27c8af7fd9b8d0386d1e92ef5e6eebc5074163157d66ef8"}
Jul 24 17:33:44.271799 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:44.271330    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2f844-9b679bb8-gzfxk" event={"ID":"f9fecf19-4c51-4a7c-8036-e64822d7ba44","Type":"ContainerStarted","Data":"fb1ddd762bc68146a4e231773164e3f3c781d7602e127a55146c9a0e96879330"}
Jul 24 17:33:44.271799 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:44.271358    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2f844-9b679bb8-gzfxk"
Jul 24 17:33:44.272674 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:44.272648    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2f844-8845757cd-289sl" event={"ID":"5f288f4c-fda7-45ea-b4a7-dfb5f463816a","Type":"ContainerStarted","Data":"b3c2d018ba8b2255edd76f2a9fa6638119f203af7d8af27c535edeff38931d7b"}
Jul 24 17:33:44.272767 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:44.272699    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2f844-8845757cd-289sl"
Jul 24 17:33:44.272767 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:44.272717    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2f844-8845757cd-289sl" event={"ID":"5f288f4c-fda7-45ea-b4a7-dfb5f463816a","Type":"ContainerStarted","Data":"56505f0a8ba3b8d13caf14ffaf257622a26b9941368b95ab7b2c6d348fbf874b"}
Jul 24 17:33:44.292237 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:44.292183    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2f844-9b679bb8-gzfxk" podStartSLOduration=2.292166703 podStartE2EDuration="2.292166703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:33:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:33:44.290191964 +0000 UTC m=+3046.932155847" watchObservedRunningTime="2026-07-24 17:33:44.292166703 +0000 UTC m=+3046.934130624"
Jul 24 17:33:44.315778 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:44.315725    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2f844-8845757cd-289sl" podStartSLOduration=2.315707104 podStartE2EDuration="2.315707104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:33:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:33:44.313638729 +0000 UTC m=+3046.955602639" watchObservedRunningTime="2026-07-24 17:33:44.315707104 +0000 UTC m=+3046.957671023"
Jul 24 17:33:46.212993 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:46.212956    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:33:55.280803 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:55.280772    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2f844-8845757cd-289sl"
Jul 24 17:33:55.281310 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:33:55.280844    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2f844-9b679bb8-gzfxk"
Jul 24 17:34:03.363340 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:03.363305    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-npzx6_669adb99-c557-4f65-95e0-af083882951f/payload-processing/0.log"
Jul 24 17:34:03.477298 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:03.477256    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a2f844-8845757cd-289sl_5f288f4c-fda7-45ea-b4a7-dfb5f463816a/payload-processing/0.log"
Jul 24 17:34:12.425476 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:12.425447    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:29.542916 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:29.542871    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:29.643873 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:29.643838    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:33.702503 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:33.702470    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:44.223536 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:44.223504    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-6a1c15-c6d459c76-zp68x_01d35db8-e624-4941-9212-78752dc2c387/payload-processing/0.log"
Jul 24 17:34:44.355120 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:44.355089    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-npzx6_669adb99-c557-4f65-95e0-af083882951f/payload-processing/0.log"
Jul 24 17:34:44.475706 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:44.475630    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a2f844-8845757cd-289sl_5f288f4c-fda7-45ea-b4a7-dfb5f463816a/payload-processing/0.log"
Jul 24 17:34:44.746448 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:44.746364    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:45.105545 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:45.105452    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:45.319451 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:45.319418    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:48.263640 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:48.263607    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:48.365417 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:48.365373    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:49.563790 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:49.563761    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:49.718390 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:49.718359    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:50.149344 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:50.149313    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:50.245453 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:50.245418    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:34:50.258201 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:50.258174    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-600cad-586ff4f8c6-lz765"]
Jul 24 17:34:50.262014 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:50.261994    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-600cad-586ff4f8c6-lz765"
Jul 24 17:34:50.265129 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:50.265110    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-600cad-dockercfg-x9l7m\""
Jul 24 17:34:50.265278 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:50.265260    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-600cad\""
Jul 24 17:34:50.271340 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:50.271316    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-600cad-586ff4f8c6-lz765"]
Jul 24 17:34:50.277582 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:50.277559    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm7hm\" (UniqueName: \"kubernetes.io/projected/8ae73fdd-bb7a-4ecf-8fc1-f3da7b17c795-kube-api-access-sm7hm\") pod \"payload-pre-processing-e2e-ipp-cleanup-600cad-586ff4f8c6-lz765\" (UID: \"8ae73fdd-bb7a-4ecf-8fc1-f3da7b17c795\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-600cad-586ff4f8c6-lz765"
Jul 24 17:34:50.277695 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:50.277596    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ae73fdd-bb7a-4ecf-8fc1-f3da7b17c795-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-600cad-586ff4f8c6-lz765\" (UID: \"8ae73fdd-bb7a-4ecf-8fc1-f3da7b17c795\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-600cad-586ff4f8c6-lz765"
Jul 24 17:34:50.286695 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:50.286661    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-600cad-6dbcf8f648-6gxdd"]
Jul 24 17:34:50.291099 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:50.291079    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-600cad-6dbcf8f648-6gxdd"
Jul 24 17:34:50.300718 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:50.300692    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-600cad-6dbcf8f648-6gxdd"]
Jul 24 17:34:50.378873 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:50.378838    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm7hm\" (UniqueName: \"kubernetes.io/projected/8ae73fdd-bb7a-4ecf-8fc1-f3da7b17c795-kube-api-access-sm7hm\") pod \"payload-pre-processing-e2e-ipp-cleanup-600cad-586ff4f8c6-lz765\" (UID: \"8ae73fdd-bb7a-4ecf-8fc1-f3da7b17c795\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-600cad-586ff4f8c6-lz765"
Jul 24 17:34:50.379080 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:50.378882    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9nj4\" (UniqueName: \"kubernetes.io/projected/ab1da480-9c5c-4176-9794-a449e32e7196-kube-api-access-s9nj4\") pod \"payload-processing-e2e-ipp-cleanup-600cad-6dbcf8f648-6gxdd\" (UID: \"ab1da480-9c5c-4176-9794-a449e32e7196\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-600cad-6dbcf8f648-6gxdd"
Jul 24 17:34:50.379080 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:50.378922    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ae73fdd-bb7a-4ecf-8fc1-f3da7b17c795-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-600cad-586ff4f8c6-lz765\" (UID: \"8ae73fdd-bb7a-4ecf-8fc1-f3da7b17c795\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-600cad-586ff4f8c6-lz765"
Jul 24 17:34:50.379207 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:50.379078    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab1da480-9c5c-4176-9794-a449e32e7196-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-600cad-6dbcf8f648-6gxdd\" (UID: \"ab1da480-9c5c-4176-9794-a449e32e7196\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-600cad-6dbcf8f648-6gxdd"
Jul 24 17:34:50.379711 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:50.379688    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ae73fdd-bb7a-4ecf-8fc1-f3da7b17c795-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-600cad-586ff4f8c6-lz765\" (UID: \"8ae73fdd-bb7a-4ecf-8fc1-f3da7b17c795\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-600cad-586ff4f8c6-lz765"
Jul 24 17:34:50.388089 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:50.388064    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm7hm\" (UniqueName: \"kubernetes.io/projected/8ae73fdd-bb7a-4ecf-8fc1-f3da7b17c795-kube-api-access-sm7hm\") pod \"payload-pre-processing-e2e-ipp-cleanup-600cad-586ff4f8c6-lz765\" (UID: \"8ae73fdd-bb7a-4ecf-8fc1-f3da7b17c795\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-600cad-586ff4f8c6-lz765"
Jul 24 17:34:50.480335 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:50.480232    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab1da480-9c5c-4176-9794-a449e32e7196-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-600cad-6dbcf8f648-6gxdd\" (UID: \"ab1da480-9c5c-4176-9794-a449e32e7196\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-600cad-6dbcf8f648-6gxdd"
Jul 24 17:34:50.480335 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:50.480322    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9nj4\" (UniqueName: \"kubernetes.io/projected/ab1da480-9c5c-4176-9794-a449e32e7196-kube-api-access-s9nj4\") pod \"payload-processing-e2e-ipp-cleanup-600cad-6dbcf8f648-6gxdd\" (UID: \"ab1da480-9c5c-4176-9794-a449e32e7196\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-600cad-6dbcf8f648-6gxdd"
Jul 24 17:34:50.481330 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:50.481307    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab1da480-9c5c-4176-9794-a449e32e7196-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-600cad-6dbcf8f648-6gxdd\" (UID: \"ab1da480-9c5c-4176-9794-a449e32e7196\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-600cad-6dbcf8f648-6gxdd"
Jul 24 17:34:50.490046 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:50.490022    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9nj4\" (UniqueName: \"kubernetes.io/projected/ab1da480-9c5c-4176-9794-a449e32e7196-kube-api-access-s9nj4\") pod \"payload-processing-e2e-ipp-cleanup-600cad-6dbcf8f648-6gxdd\" (UID: \"ab1da480-9c5c-4176-9794-a449e32e7196\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-600cad-6dbcf8f648-6gxdd"
Jul 24 17:34:50.573917 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:50.573875    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-600cad-586ff4f8c6-lz765"
Jul 24 17:34:50.604443 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:50.604412    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-600cad-6dbcf8f648-6gxdd"
Jul 24 17:34:50.737081 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:50.737003    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-600cad-586ff4f8c6-lz765"]
Jul 24 17:34:50.741867 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:34:50.741765    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ae73fdd_bb7a_4ecf_8fc1_f3da7b17c795.slice/crio-482734e4063716355cdbbf77a8bb318d9271e6cfb40b366bf9305fc80d42299b WatchSource:0}: Error finding container 482734e4063716355cdbbf77a8bb318d9271e6cfb40b366bf9305fc80d42299b: Status 404 returned error can't find the container with id 482734e4063716355cdbbf77a8bb318d9271e6cfb40b366bf9305fc80d42299b
Jul 24 17:34:50.970122 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:50.970076    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-600cad-6dbcf8f648-6gxdd"]
Jul 24 17:34:50.972432 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:34:50.972400    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab1da480_9c5c_4176_9794_a449e32e7196.slice/crio-26f9f04a7d0fd3cfbf5a8a307732db62d359d0871f97a30648ead120e63e8757 WatchSource:0}: Error finding container 26f9f04a7d0fd3cfbf5a8a307732db62d359d0871f97a30648ead120e63e8757: Status 404 returned error can't find the container with id 26f9f04a7d0fd3cfbf5a8a307732db62d359d0871f97a30648ead120e63e8757
Jul 24 17:34:51.635843 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:51.635803    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-600cad-6dbcf8f648-6gxdd" event={"ID":"ab1da480-9c5c-4176-9794-a449e32e7196","Type":"ContainerStarted","Data":"70e63896f3707614fb43f266aed2e987fea8a715b0e7099ef62bac00c6879a1f"}
Jul 24 17:34:51.636339 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:51.635880    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-600cad-6dbcf8f648-6gxdd"
Jul 24 17:34:51.636339 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:51.635906    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-600cad-6dbcf8f648-6gxdd" event={"ID":"ab1da480-9c5c-4176-9794-a449e32e7196","Type":"ContainerStarted","Data":"26f9f04a7d0fd3cfbf5a8a307732db62d359d0871f97a30648ead120e63e8757"}
Jul 24 17:34:51.637446 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:51.637421    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-600cad-586ff4f8c6-lz765" event={"ID":"8ae73fdd-bb7a-4ecf-8fc1-f3da7b17c795","Type":"ContainerStarted","Data":"d074b5c66bfaebdcbe16e36da604c9beb5b3bb3d11f7b8363b033a460dd19cdc"}
Jul 24 17:34:51.637565 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:51.637458    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-600cad-586ff4f8c6-lz765" event={"ID":"8ae73fdd-bb7a-4ecf-8fc1-f3da7b17c795","Type":"ContainerStarted","Data":"482734e4063716355cdbbf77a8bb318d9271e6cfb40b366bf9305fc80d42299b"}
Jul 24 17:34:51.637565 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:51.637477    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-600cad-586ff4f8c6-lz765"
Jul 24 17:34:51.654527 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:51.654475    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-600cad-6dbcf8f648-6gxdd" podStartSLOduration=1.654457488 podStartE2EDuration="1.654457488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:34:51.651568649 +0000 UTC m=+3114.293532549" watchObservedRunningTime="2026-07-24 17:34:51.654457488 +0000 UTC m=+3114.296421375"
Jul 24 17:34:51.673959 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:51.673884    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-600cad-586ff4f8c6-lz765" podStartSLOduration=1.6738663950000001 podStartE2EDuration="1.673866395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:34:51.672806098 +0000 UTC m=+3114.314770001" watchObservedRunningTime="2026-07-24 17:34:51.673866395 +0000 UTC m=+3114.315830279"
Jul 24 17:34:52.574879 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:34:52.574844    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:35:02.646431 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:02.646391    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-600cad-6dbcf8f648-6gxdd"
Jul 24 17:35:02.646927 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:02.646472    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-600cad-586ff4f8c6-lz765"
Jul 24 17:35:02.962281 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:02.962196    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:35:03.063013 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:03.062977    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:35:15.576256 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:15.576221    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:35:15.597845 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:15.597811    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-600cad-6dbcf8f648-6gxdd"]
Jul 24 17:35:15.598095 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:15.598067    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-600cad-6dbcf8f648-6gxdd" podUID="ab1da480-9c5c-4176-9794-a449e32e7196" containerName="payload-processing" containerID="cri-o://70e63896f3707614fb43f266aed2e987fea8a715b0e7099ef62bac00c6879a1f" gracePeriod=30
Jul 24 17:35:15.604688 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:15.604655    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-600cad-586ff4f8c6-lz765"]
Jul 24 17:35:15.604911 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:15.604879    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-600cad-586ff4f8c6-lz765" podUID="8ae73fdd-bb7a-4ecf-8fc1-f3da7b17c795" containerName="payload-pre-processing" containerID="cri-o://d074b5c66bfaebdcbe16e36da604c9beb5b3bb3d11f7b8363b033a460dd19cdc" gracePeriod=30
Jul 24 17:35:15.677993 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:15.677966    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:35:15.774076 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:15.773963    2563 generic.go:356] "Generic (PLEG): container finished" podID="ab1da480-9c5c-4176-9794-a449e32e7196" containerID="70e63896f3707614fb43f266aed2e987fea8a715b0e7099ef62bac00c6879a1f" exitCode=0
Jul 24 17:35:15.774076 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:15.774000    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-600cad-6dbcf8f648-6gxdd" event={"ID":"ab1da480-9c5c-4176-9794-a449e32e7196","Type":"ContainerDied","Data":"70e63896f3707614fb43f266aed2e987fea8a715b0e7099ef62bac00c6879a1f"}
Jul 24 17:35:15.776475 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:15.776440    2563 generic.go:356] "Generic (PLEG): container finished" podID="8ae73fdd-bb7a-4ecf-8fc1-f3da7b17c795" containerID="d074b5c66bfaebdcbe16e36da604c9beb5b3bb3d11f7b8363b033a460dd19cdc" exitCode=0
Jul 24 17:35:15.776583 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:15.776479    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-600cad-586ff4f8c6-lz765" event={"ID":"8ae73fdd-bb7a-4ecf-8fc1-f3da7b17c795","Type":"ContainerDied","Data":"d074b5c66bfaebdcbe16e36da604c9beb5b3bb3d11f7b8363b033a460dd19cdc"}
Jul 24 17:35:15.875717 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:15.875689    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-600cad-6dbcf8f648-6gxdd"
Jul 24 17:35:15.894007 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:15.893984    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-600cad-586ff4f8c6-lz765"
Jul 24 17:35:16.028252 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:16.028220    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9nj4\" (UniqueName: \"kubernetes.io/projected/ab1da480-9c5c-4176-9794-a449e32e7196-kube-api-access-s9nj4\") pod \"ab1da480-9c5c-4176-9794-a449e32e7196\" (UID: \"ab1da480-9c5c-4176-9794-a449e32e7196\") "
Jul 24 17:35:16.028416 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:16.028259    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ae73fdd-bb7a-4ecf-8fc1-f3da7b17c795-plugins-config-volume\") pod \"8ae73fdd-bb7a-4ecf-8fc1-f3da7b17c795\" (UID: \"8ae73fdd-bb7a-4ecf-8fc1-f3da7b17c795\") "
Jul 24 17:35:16.028416 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:16.028285    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sm7hm\" (UniqueName: \"kubernetes.io/projected/8ae73fdd-bb7a-4ecf-8fc1-f3da7b17c795-kube-api-access-sm7hm\") pod \"8ae73fdd-bb7a-4ecf-8fc1-f3da7b17c795\" (UID: \"8ae73fdd-bb7a-4ecf-8fc1-f3da7b17c795\") "
Jul 24 17:35:16.028416 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:16.028316    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab1da480-9c5c-4176-9794-a449e32e7196-plugins-config-volume\") pod \"ab1da480-9c5c-4176-9794-a449e32e7196\" (UID: \"ab1da480-9c5c-4176-9794-a449e32e7196\") "
Jul 24 17:35:16.028646 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:16.028620    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ae73fdd-bb7a-4ecf-8fc1-f3da7b17c795-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8ae73fdd-bb7a-4ecf-8fc1-f3da7b17c795" (UID: "8ae73fdd-bb7a-4ecf-8fc1-f3da7b17c795"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:35:16.028736 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:16.028715    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab1da480-9c5c-4176-9794-a449e32e7196-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ab1da480-9c5c-4176-9794-a449e32e7196" (UID: "ab1da480-9c5c-4176-9794-a449e32e7196"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:35:16.030342 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:16.030320    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab1da480-9c5c-4176-9794-a449e32e7196-kube-api-access-s9nj4" (OuterVolumeSpecName: "kube-api-access-s9nj4") pod "ab1da480-9c5c-4176-9794-a449e32e7196" (UID: "ab1da480-9c5c-4176-9794-a449e32e7196"). InnerVolumeSpecName "kube-api-access-s9nj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:35:16.030471 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:16.030452    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ae73fdd-bb7a-4ecf-8fc1-f3da7b17c795-kube-api-access-sm7hm" (OuterVolumeSpecName: "kube-api-access-sm7hm") pod "8ae73fdd-bb7a-4ecf-8fc1-f3da7b17c795" (UID: "8ae73fdd-bb7a-4ecf-8fc1-f3da7b17c795"). InnerVolumeSpecName "kube-api-access-sm7hm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:35:16.129764 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:16.129685    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab1da480-9c5c-4176-9794-a449e32e7196-plugins-config-volume\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:35:16.129764 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:16.129710    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9nj4\" (UniqueName: \"kubernetes.io/projected/ab1da480-9c5c-4176-9794-a449e32e7196-kube-api-access-s9nj4\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:35:16.129764 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:16.129725    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ae73fdd-bb7a-4ecf-8fc1-f3da7b17c795-plugins-config-volume\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:35:16.129764 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:16.129737    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm7hm\" (UniqueName: \"kubernetes.io/projected/8ae73fdd-bb7a-4ecf-8fc1-f3da7b17c795-kube-api-access-sm7hm\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:35:16.782792 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:16.782767    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-600cad-6dbcf8f648-6gxdd"
Jul 24 17:35:16.783256 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:16.782819    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-600cad-6dbcf8f648-6gxdd" event={"ID":"ab1da480-9c5c-4176-9794-a449e32e7196","Type":"ContainerDied","Data":"26f9f04a7d0fd3cfbf5a8a307732db62d359d0871f97a30648ead120e63e8757"}
Jul 24 17:35:16.783256 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:16.782880    2563 scope.go:117] "RemoveContainer" containerID="70e63896f3707614fb43f266aed2e987fea8a715b0e7099ef62bac00c6879a1f"
Jul 24 17:35:16.784411 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:16.784391    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-600cad-586ff4f8c6-lz765" event={"ID":"8ae73fdd-bb7a-4ecf-8fc1-f3da7b17c795","Type":"ContainerDied","Data":"482734e4063716355cdbbf77a8bb318d9271e6cfb40b366bf9305fc80d42299b"}
Jul 24 17:35:16.784497 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:16.784421    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-600cad-586ff4f8c6-lz765"
Jul 24 17:35:16.794297 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:16.794271    2563 scope.go:117] "RemoveContainer" containerID="d074b5c66bfaebdcbe16e36da604c9beb5b3bb3d11f7b8363b033a460dd19cdc"
Jul 24 17:35:16.806285 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:16.806261    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-600cad-6dbcf8f648-6gxdd"]
Jul 24 17:35:16.809981 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:16.809959    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-600cad-6dbcf8f648-6gxdd"]
Jul 24 17:35:16.825727 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:16.820998    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-600cad-586ff4f8c6-lz765"]
Jul 24 17:35:16.828238 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:16.828214    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-600cad-586ff4f8c6-lz765"]
Jul 24 17:35:18.003677 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:18.003643    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ae73fdd-bb7a-4ecf-8fc1-f3da7b17c795" path="/var/lib/kubelet/pods/8ae73fdd-bb7a-4ecf-8fc1-f3da7b17c795/volumes"
Jul 24 17:35:18.004072 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:18.004030    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab1da480-9c5c-4176-9794-a449e32e7196" path="/var/lib/kubelet/pods/ab1da480-9c5c-4176-9794-a449e32e7196/volumes"
Jul 24 17:35:18.593002 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:18.592965    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:35:19.928887 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:19.928857    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:35:25.239726 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:25.239695    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:35:25.349773 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:25.349735    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:35:25.529882 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:25.529798    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:35:25.767879 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:25.767850    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:35:45.235848 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:45.235807    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:35:45.267540 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:45.267507    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6a1c15-c6d459c76-zp68x"]
Jul 24 17:35:45.267810 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:45.267768    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a1c15-c6d459c76-zp68x" podUID="01d35db8-e624-4941-9212-78752dc2c387" containerName="payload-processing" containerID="cri-o://d84c2af317e2f3bac6c03527d2090036c27af17ff0950a3d69cc1572155c4b05" gracePeriod=30
Jul 24 17:35:45.272692 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:45.272664    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6a1c15-6d5f5b75bd-crz8r"]
Jul 24 17:35:45.273399 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:45.272924    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a1c15-6d5f5b75bd-crz8r" podUID="69a0d029-1031-4849-9060-ff5fb0e35065" containerName="payload-pre-processing" containerID="cri-o://97cd71a094bb3f40cfe3f85c6dba6dd40bd07add6f1117d6a948629ed831abbc" gracePeriod=30
Jul 24 17:35:45.307463 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:45.307396    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:35:45.366789 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:45.366757    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:35:45.578212 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:45.578187    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a1c15-c6d459c76-zp68x"
Jul 24 17:35:45.581296 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:45.581278    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a1c15-6d5f5b75bd-crz8r"
Jul 24 17:35:45.692866 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:45.692839    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69a0d029-1031-4849-9060-ff5fb0e35065-plugins-config-volume\") pod \"69a0d029-1031-4849-9060-ff5fb0e35065\" (UID: \"69a0d029-1031-4849-9060-ff5fb0e35065\") "
Jul 24 17:35:45.692866 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:45.692870    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01d35db8-e624-4941-9212-78752dc2c387-plugins-config-volume\") pod \"01d35db8-e624-4941-9212-78752dc2c387\" (UID: \"01d35db8-e624-4941-9212-78752dc2c387\") "
Jul 24 17:35:45.693135 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:45.692913    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shtpb\" (UniqueName: \"kubernetes.io/projected/69a0d029-1031-4849-9060-ff5fb0e35065-kube-api-access-shtpb\") pod \"69a0d029-1031-4849-9060-ff5fb0e35065\" (UID: \"69a0d029-1031-4849-9060-ff5fb0e35065\") "
Jul 24 17:35:45.693135 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:45.693000    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcq54\" (UniqueName: \"kubernetes.io/projected/01d35db8-e624-4941-9212-78752dc2c387-kube-api-access-lcq54\") pod \"01d35db8-e624-4941-9212-78752dc2c387\" (UID: \"01d35db8-e624-4941-9212-78752dc2c387\") "
Jul 24 17:35:45.693345 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:45.693319    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01d35db8-e624-4941-9212-78752dc2c387-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "01d35db8-e624-4941-9212-78752dc2c387" (UID: "01d35db8-e624-4941-9212-78752dc2c387"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:35:45.693404 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:45.693322    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69a0d029-1031-4849-9060-ff5fb0e35065-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "69a0d029-1031-4849-9060-ff5fb0e35065" (UID: "69a0d029-1031-4849-9060-ff5fb0e35065"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:35:45.694990 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:45.694966    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01d35db8-e624-4941-9212-78752dc2c387-kube-api-access-lcq54" (OuterVolumeSpecName: "kube-api-access-lcq54") pod "01d35db8-e624-4941-9212-78752dc2c387" (UID: "01d35db8-e624-4941-9212-78752dc2c387"). InnerVolumeSpecName "kube-api-access-lcq54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:35:45.695056 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:45.695006    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a0d029-1031-4849-9060-ff5fb0e35065-kube-api-access-shtpb" (OuterVolumeSpecName: "kube-api-access-shtpb") pod "69a0d029-1031-4849-9060-ff5fb0e35065" (UID: "69a0d029-1031-4849-9060-ff5fb0e35065"). InnerVolumeSpecName "kube-api-access-shtpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:35:45.794446 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:45.794380    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69a0d029-1031-4849-9060-ff5fb0e35065-plugins-config-volume\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:35:45.794446 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:45.794406    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01d35db8-e624-4941-9212-78752dc2c387-plugins-config-volume\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:35:45.794446 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:45.794418    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shtpb\" (UniqueName: \"kubernetes.io/projected/69a0d029-1031-4849-9060-ff5fb0e35065-kube-api-access-shtpb\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:35:45.794446 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:45.794427    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcq54\" (UniqueName: \"kubernetes.io/projected/01d35db8-e624-4941-9212-78752dc2c387-kube-api-access-lcq54\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:35:45.944755 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:45.944723    2563 generic.go:356] "Generic (PLEG): container finished" podID="69a0d029-1031-4849-9060-ff5fb0e35065" containerID="97cd71a094bb3f40cfe3f85c6dba6dd40bd07add6f1117d6a948629ed831abbc" exitCode=0
Jul 24 17:35:45.944968 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:45.944769    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a1c15-6d5f5b75bd-crz8r"
Jul 24 17:35:45.944968 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:45.944805    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a1c15-6d5f5b75bd-crz8r" event={"ID":"69a0d029-1031-4849-9060-ff5fb0e35065","Type":"ContainerDied","Data":"97cd71a094bb3f40cfe3f85c6dba6dd40bd07add6f1117d6a948629ed831abbc"}
Jul 24 17:35:45.944968 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:45.944874    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a1c15-6d5f5b75bd-crz8r" event={"ID":"69a0d029-1031-4849-9060-ff5fb0e35065","Type":"ContainerDied","Data":"9189aefb3121a0a52e4caf3f7585abfb7bf461766d664bff0465ca29393c8126"}
Jul 24 17:35:45.944968 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:45.944888    2563 scope.go:117] "RemoveContainer" containerID="97cd71a094bb3f40cfe3f85c6dba6dd40bd07add6f1117d6a948629ed831abbc"
Jul 24 17:35:45.946271 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:45.946246    2563 generic.go:356] "Generic (PLEG): container finished" podID="01d35db8-e624-4941-9212-78752dc2c387" containerID="d84c2af317e2f3bac6c03527d2090036c27af17ff0950a3d69cc1572155c4b05" exitCode=0
Jul 24 17:35:45.946402 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:45.946294    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a1c15-c6d459c76-zp68x"
Jul 24 17:35:45.946402 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:45.946308    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a1c15-c6d459c76-zp68x" event={"ID":"01d35db8-e624-4941-9212-78752dc2c387","Type":"ContainerDied","Data":"d84c2af317e2f3bac6c03527d2090036c27af17ff0950a3d69cc1572155c4b05"}
Jul 24 17:35:45.946402 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:45.946345    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a1c15-c6d459c76-zp68x" event={"ID":"01d35db8-e624-4941-9212-78752dc2c387","Type":"ContainerDied","Data":"aa9a15f0b6ddfdd559720b87a7a892d38cede6098c8835f91424190565772c55"}
Jul 24 17:35:45.955132 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:45.954970    2563 scope.go:117] "RemoveContainer" containerID="97cd71a094bb3f40cfe3f85c6dba6dd40bd07add6f1117d6a948629ed831abbc"
Jul 24 17:35:45.955303 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:35:45.955277    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97cd71a094bb3f40cfe3f85c6dba6dd40bd07add6f1117d6a948629ed831abbc\": container with ID starting with 97cd71a094bb3f40cfe3f85c6dba6dd40bd07add6f1117d6a948629ed831abbc not found: ID does not exist" containerID="97cd71a094bb3f40cfe3f85c6dba6dd40bd07add6f1117d6a948629ed831abbc"
Jul 24 17:35:45.955387 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:45.955313    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97cd71a094bb3f40cfe3f85c6dba6dd40bd07add6f1117d6a948629ed831abbc"} err="failed to get container status \"97cd71a094bb3f40cfe3f85c6dba6dd40bd07add6f1117d6a948629ed831abbc\": rpc error: code = NotFound desc = could not find container \"97cd71a094bb3f40cfe3f85c6dba6dd40bd07add6f1117d6a948629ed831abbc\": container with ID starting with 97cd71a094bb3f40cfe3f85c6dba6dd40bd07add6f1117d6a948629ed831abbc not found: ID does not exist"
Jul 24 17:35:45.955387 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:45.955337    2563 scope.go:117] "RemoveContainer" containerID="d84c2af317e2f3bac6c03527d2090036c27af17ff0950a3d69cc1572155c4b05"
Jul 24 17:35:45.963226 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:45.963212    2563 scope.go:117] "RemoveContainer" containerID="d84c2af317e2f3bac6c03527d2090036c27af17ff0950a3d69cc1572155c4b05"
Jul 24 17:35:45.963464 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:35:45.963448    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d84c2af317e2f3bac6c03527d2090036c27af17ff0950a3d69cc1572155c4b05\": container with ID starting with d84c2af317e2f3bac6c03527d2090036c27af17ff0950a3d69cc1572155c4b05 not found: ID does not exist" containerID="d84c2af317e2f3bac6c03527d2090036c27af17ff0950a3d69cc1572155c4b05"
Jul 24 17:35:45.963513 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:45.963470    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d84c2af317e2f3bac6c03527d2090036c27af17ff0950a3d69cc1572155c4b05"} err="failed to get container status \"d84c2af317e2f3bac6c03527d2090036c27af17ff0950a3d69cc1572155c4b05\": rpc error: code = NotFound desc = could not find container \"d84c2af317e2f3bac6c03527d2090036c27af17ff0950a3d69cc1572155c4b05\": container with ID starting with d84c2af317e2f3bac6c03527d2090036c27af17ff0950a3d69cc1572155c4b05 not found: ID does not exist"
Jul 24 17:35:45.971840 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:45.971817    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6a1c15-c6d459c76-zp68x"]
Jul 24 17:35:45.974253 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:45.974235    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6a1c15-c6d459c76-zp68x"]
Jul 24 17:35:45.985246 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:45.985222    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6a1c15-6d5f5b75bd-crz8r"]
Jul 24 17:35:45.993397 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:45.993367    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6a1c15-6d5f5b75bd-crz8r"]
Jul 24 17:35:46.002357 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:46.002335    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01d35db8-e624-4941-9212-78752dc2c387" path="/var/lib/kubelet/pods/01d35db8-e624-4941-9212-78752dc2c387/volumes"
Jul 24 17:35:46.002707 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:46.002694    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69a0d029-1031-4849-9060-ff5fb0e35065" path="/var/lib/kubelet/pods/69a0d029-1031-4849-9060-ff5fb0e35065/volumes"
Jul 24 17:35:49.256416 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:49.256383    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:35:49.643390 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:49.643359    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:35:54.730112 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:35:54.730078    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:36:30.914781 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:30.914751    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:36:31.977950 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:31.977915    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:36:37.550860 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:37.550820    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:36:37.622140 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:37.622108    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:36:37.932821 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:37.932786    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:36:38.101430 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:38.101398    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:36:48.401462 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:48.401426    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:36:48.421980 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:48.421925    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a2f844-8845757cd-289sl"]
Jul 24 17:36:48.422177 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:48.422154    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2f844-8845757cd-289sl" podUID="5f288f4c-fda7-45ea-b4a7-dfb5f463816a" containerName="payload-processing" containerID="cri-o://b3c2d018ba8b2255edd76f2a9fa6638119f203af7d8af27c535edeff38931d7b" gracePeriod=30
Jul 24 17:36:48.433634 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:48.433608    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a2f844-9b679bb8-gzfxk"]
Jul 24 17:36:48.433874 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:48.433829    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2f844-9b679bb8-gzfxk" podUID="f9fecf19-4c51-4a7c-8036-e64822d7ba44" containerName="payload-pre-processing" containerID="cri-o://3bc0810e92eb7c26d27c8af7fd9b8d0386d1e92ef5e6eebc5074163157d66ef8" gracePeriod=30
Jul 24 17:36:48.469191 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:48.469156    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:36:48.533684 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:48.533659    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:36:48.719098 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:48.719071    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2f844-8845757cd-289sl"
Jul 24 17:36:48.722368 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:48.722349    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2f844-9b679bb8-gzfxk"
Jul 24 17:36:48.854115 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:48.854083    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9fecf19-4c51-4a7c-8036-e64822d7ba44-plugins-config-volume\") pod \"f9fecf19-4c51-4a7c-8036-e64822d7ba44\" (UID: \"f9fecf19-4c51-4a7c-8036-e64822d7ba44\") "
Jul 24 17:36:48.854328 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:48.854144    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzcpg\" (UniqueName: \"kubernetes.io/projected/f9fecf19-4c51-4a7c-8036-e64822d7ba44-kube-api-access-dzcpg\") pod \"f9fecf19-4c51-4a7c-8036-e64822d7ba44\" (UID: \"f9fecf19-4c51-4a7c-8036-e64822d7ba44\") "
Jul 24 17:36:48.854328 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:48.854187    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f288f4c-fda7-45ea-b4a7-dfb5f463816a-plugins-config-volume\") pod \"5f288f4c-fda7-45ea-b4a7-dfb5f463816a\" (UID: \"5f288f4c-fda7-45ea-b4a7-dfb5f463816a\") "
Jul 24 17:36:48.854328 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:48.854210    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mc482\" (UniqueName: \"kubernetes.io/projected/5f288f4c-fda7-45ea-b4a7-dfb5f463816a-kube-api-access-mc482\") pod \"5f288f4c-fda7-45ea-b4a7-dfb5f463816a\" (UID: \"5f288f4c-fda7-45ea-b4a7-dfb5f463816a\") "
Jul 24 17:36:48.854655 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:48.854624    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9fecf19-4c51-4a7c-8036-e64822d7ba44-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f9fecf19-4c51-4a7c-8036-e64822d7ba44" (UID: "f9fecf19-4c51-4a7c-8036-e64822d7ba44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:36:48.854655 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:48.854628    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f288f4c-fda7-45ea-b4a7-dfb5f463816a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5f288f4c-fda7-45ea-b4a7-dfb5f463816a" (UID: "5f288f4c-fda7-45ea-b4a7-dfb5f463816a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:36:48.856440 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:48.856420    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9fecf19-4c51-4a7c-8036-e64822d7ba44-kube-api-access-dzcpg" (OuterVolumeSpecName: "kube-api-access-dzcpg") pod "f9fecf19-4c51-4a7c-8036-e64822d7ba44" (UID: "f9fecf19-4c51-4a7c-8036-e64822d7ba44"). InnerVolumeSpecName "kube-api-access-dzcpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:36:48.856589 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:48.856570    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f288f4c-fda7-45ea-b4a7-dfb5f463816a-kube-api-access-mc482" (OuterVolumeSpecName: "kube-api-access-mc482") pod "5f288f4c-fda7-45ea-b4a7-dfb5f463816a" (UID: "5f288f4c-fda7-45ea-b4a7-dfb5f463816a"). InnerVolumeSpecName "kube-api-access-mc482". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:36:48.955402 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:48.955329    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9fecf19-4c51-4a7c-8036-e64822d7ba44-plugins-config-volume\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:36:48.955402 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:48.955353    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzcpg\" (UniqueName: \"kubernetes.io/projected/f9fecf19-4c51-4a7c-8036-e64822d7ba44-kube-api-access-dzcpg\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:36:48.955402 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:48.955364    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f288f4c-fda7-45ea-b4a7-dfb5f463816a-plugins-config-volume\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:36:48.955402 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:48.955374    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc482\" (UniqueName: \"kubernetes.io/projected/5f288f4c-fda7-45ea-b4a7-dfb5f463816a-kube-api-access-mc482\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:36:49.275260 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:49.275176    2563 generic.go:356] "Generic (PLEG): container finished" podID="f9fecf19-4c51-4a7c-8036-e64822d7ba44" containerID="3bc0810e92eb7c26d27c8af7fd9b8d0386d1e92ef5e6eebc5074163157d66ef8" exitCode=0
Jul 24 17:36:49.275260 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:49.275238    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2f844-9b679bb8-gzfxk"
Jul 24 17:36:49.275497 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:49.275258    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2f844-9b679bb8-gzfxk" event={"ID":"f9fecf19-4c51-4a7c-8036-e64822d7ba44","Type":"ContainerDied","Data":"3bc0810e92eb7c26d27c8af7fd9b8d0386d1e92ef5e6eebc5074163157d66ef8"}
Jul 24 17:36:49.275497 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:49.275310    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2f844-9b679bb8-gzfxk" event={"ID":"f9fecf19-4c51-4a7c-8036-e64822d7ba44","Type":"ContainerDied","Data":"fb1ddd762bc68146a4e231773164e3f3c781d7602e127a55146c9a0e96879330"}
Jul 24 17:36:49.275497 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:49.275336    2563 scope.go:117] "RemoveContainer" containerID="3bc0810e92eb7c26d27c8af7fd9b8d0386d1e92ef5e6eebc5074163157d66ef8"
Jul 24 17:36:49.276858 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:49.276831    2563 generic.go:356] "Generic (PLEG): container finished" podID="5f288f4c-fda7-45ea-b4a7-dfb5f463816a" containerID="b3c2d018ba8b2255edd76f2a9fa6638119f203af7d8af27c535edeff38931d7b" exitCode=0
Jul 24 17:36:49.276997 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:49.276883    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2f844-8845757cd-289sl" event={"ID":"5f288f4c-fda7-45ea-b4a7-dfb5f463816a","Type":"ContainerDied","Data":"b3c2d018ba8b2255edd76f2a9fa6638119f203af7d8af27c535edeff38931d7b"}
Jul 24 17:36:49.276997 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:49.276925    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2f844-8845757cd-289sl" event={"ID":"5f288f4c-fda7-45ea-b4a7-dfb5f463816a","Type":"ContainerDied","Data":"56505f0a8ba3b8d13caf14ffaf257622a26b9941368b95ab7b2c6d348fbf874b"}
Jul 24 17:36:49.276997 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:49.276884    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2f844-8845757cd-289sl"
Jul 24 17:36:49.285507 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:49.285494    2563 scope.go:117] "RemoveContainer" containerID="3bc0810e92eb7c26d27c8af7fd9b8d0386d1e92ef5e6eebc5074163157d66ef8"
Jul 24 17:36:49.285738 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:36:49.285721    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bc0810e92eb7c26d27c8af7fd9b8d0386d1e92ef5e6eebc5074163157d66ef8\": container with ID starting with 3bc0810e92eb7c26d27c8af7fd9b8d0386d1e92ef5e6eebc5074163157d66ef8 not found: ID does not exist" containerID="3bc0810e92eb7c26d27c8af7fd9b8d0386d1e92ef5e6eebc5074163157d66ef8"
Jul 24 17:36:49.285799 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:49.285744    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bc0810e92eb7c26d27c8af7fd9b8d0386d1e92ef5e6eebc5074163157d66ef8"} err="failed to get container status \"3bc0810e92eb7c26d27c8af7fd9b8d0386d1e92ef5e6eebc5074163157d66ef8\": rpc error: code = NotFound desc = could not find container \"3bc0810e92eb7c26d27c8af7fd9b8d0386d1e92ef5e6eebc5074163157d66ef8\": container with ID starting with 3bc0810e92eb7c26d27c8af7fd9b8d0386d1e92ef5e6eebc5074163157d66ef8 not found: ID does not exist"
Jul 24 17:36:49.285799 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:49.285759    2563 scope.go:117] "RemoveContainer" containerID="b3c2d018ba8b2255edd76f2a9fa6638119f203af7d8af27c535edeff38931d7b"
Jul 24 17:36:49.294175 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:49.294155    2563 scope.go:117] "RemoveContainer" containerID="b3c2d018ba8b2255edd76f2a9fa6638119f203af7d8af27c535edeff38931d7b"
Jul 24 17:36:49.294440 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:36:49.294404    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3c2d018ba8b2255edd76f2a9fa6638119f203af7d8af27c535edeff38931d7b\": container with ID starting with b3c2d018ba8b2255edd76f2a9fa6638119f203af7d8af27c535edeff38931d7b not found: ID does not exist" containerID="b3c2d018ba8b2255edd76f2a9fa6638119f203af7d8af27c535edeff38931d7b"
Jul 24 17:36:49.294500 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:49.294431    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3c2d018ba8b2255edd76f2a9fa6638119f203af7d8af27c535edeff38931d7b"} err="failed to get container status \"b3c2d018ba8b2255edd76f2a9fa6638119f203af7d8af27c535edeff38931d7b\": rpc error: code = NotFound desc = could not find container \"b3c2d018ba8b2255edd76f2a9fa6638119f203af7d8af27c535edeff38931d7b\": container with ID starting with b3c2d018ba8b2255edd76f2a9fa6638119f203af7d8af27c535edeff38931d7b not found: ID does not exist"
Jul 24 17:36:49.297443 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:49.297422    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a2f844-9b679bb8-gzfxk"]
Jul 24 17:36:49.303492 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:49.303472    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a2f844-9b679bb8-gzfxk"]
Jul 24 17:36:49.314283 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:49.314259    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a2f844-8845757cd-289sl"]
Jul 24 17:36:49.320318 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:49.320299    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a2f844-8845757cd-289sl"]
Jul 24 17:36:50.002224 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:50.002191    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f288f4c-fda7-45ea-b4a7-dfb5f463816a" path="/var/lib/kubelet/pods/5f288f4c-fda7-45ea-b4a7-dfb5f463816a/volumes"
Jul 24 17:36:50.002589 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:50.002545    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9fecf19-4c51-4a7c-8036-e64822d7ba44" path="/var/lib/kubelet/pods/f9fecf19-4c51-4a7c-8036-e64822d7ba44/volumes"
Jul 24 17:36:53.317179 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:53.317149    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:36:53.723857 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:53.723806    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:36:59.536332 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:59.536298    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:36:59.640347 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:36:59.640312    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:00.007098 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:00.007043    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:00.602058 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:00.602027    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:11.339728 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:11.339684    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-de68c7-79ccd89d4f-sxqrz"]
Jul 24 17:37:11.340345 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:11.339898    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-de68c7-79ccd89d4f-sxqrz" podUID="01094640-b811-412f-b9bb-2d3cc7900efa" containerName="maas-api" containerID="cri-o://93e31cd51ab19ee6bc05b590cd6643d28811ffbd4e0b20ad0122622f0fd9a7b8" gracePeriod=30
Jul 24 17:37:11.399345 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:11.399301    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-de68c7-676cb7d874-q76wx"]
Jul 24 17:37:11.399579 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:11.399538    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-de68c7-676cb7d874-q76wx" podUID="deb9c86a-2b44-474f-8edd-a7e0183e137b" containerName="payload-processing" containerID="cri-o://7788c7de6b943929be64e3c9f1b8b7f62a1989506050cc8767333be4fff5d301" gracePeriod=30
Jul 24 17:37:11.409480 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:11.409340    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-de68c7-7b7c5fd57b-dj4g8"]
Jul 24 17:37:11.410440 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:11.410109    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-de68c7-7b7c5fd57b-dj4g8" podUID="402d7ca3-45e0-46c7-a954-050a4f86a6ed" containerName="payload-pre-processing" containerID="cri-o://3394a5c8058c6a0d6a205c59aba3fb4cfb94e90748a7eb66104d5100597d4bd9" gracePeriod=30
Jul 24 17:37:11.690346 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:11.690314    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-de68c7-79ccd89d4f-sxqrz"
Jul 24 17:37:11.707725 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:11.707696    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-de68c7-676cb7d874-q76wx"
Jul 24 17:37:11.718324 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:11.718295    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-de68c7-7b7c5fd57b-dj4g8"
Jul 24 17:37:11.749011 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:11.748978    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nr68\" (UniqueName: \"kubernetes.io/projected/01094640-b811-412f-b9bb-2d3cc7900efa-kube-api-access-5nr68\") pod \"01094640-b811-412f-b9bb-2d3cc7900efa\" (UID: \"01094640-b811-412f-b9bb-2d3cc7900efa\") "
Jul 24 17:37:11.749171 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:11.749053    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwxc2\" (UniqueName: \"kubernetes.io/projected/deb9c86a-2b44-474f-8edd-a7e0183e137b-kube-api-access-nwxc2\") pod \"deb9c86a-2b44-474f-8edd-a7e0183e137b\" (UID: \"deb9c86a-2b44-474f-8edd-a7e0183e137b\") "
Jul 24 17:37:11.749171 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:11.749073    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01094640-b811-412f-b9bb-2d3cc7900efa-maas-api-tls\") pod \"01094640-b811-412f-b9bb-2d3cc7900efa\" (UID: \"01094640-b811-412f-b9bb-2d3cc7900efa\") "
Jul 24 17:37:11.749171 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:11.749094    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01094640-b811-412f-b9bb-2d3cc7900efa-openshift-service-ca\") pod \"01094640-b811-412f-b9bb-2d3cc7900efa\" (UID: \"01094640-b811-412f-b9bb-2d3cc7900efa\") "
Jul 24 17:37:11.749171 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:11.749108    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/402d7ca3-45e0-46c7-a954-050a4f86a6ed-plugins-config-volume\") pod \"402d7ca3-45e0-46c7-a954-050a4f86a6ed\" (UID: \"402d7ca3-45e0-46c7-a954-050a4f86a6ed\") "
Jul 24 17:37:11.749171 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:11.749145    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/deb9c86a-2b44-474f-8edd-a7e0183e137b-plugins-config-volume\") pod \"deb9c86a-2b44-474f-8edd-a7e0183e137b\" (UID: \"deb9c86a-2b44-474f-8edd-a7e0183e137b\") "
Jul 24 17:37:11.749435 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:11.749178    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5qs4\" (UniqueName: \"kubernetes.io/projected/402d7ca3-45e0-46c7-a954-050a4f86a6ed-kube-api-access-r5qs4\") pod \"402d7ca3-45e0-46c7-a954-050a4f86a6ed\" (UID: \"402d7ca3-45e0-46c7-a954-050a4f86a6ed\") "
Jul 24 17:37:11.749594 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:11.749471    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01094640-b811-412f-b9bb-2d3cc7900efa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "01094640-b811-412f-b9bb-2d3cc7900efa" (UID: "01094640-b811-412f-b9bb-2d3cc7900efa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:37:11.749677 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:11.749592    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/402d7ca3-45e0-46c7-a954-050a4f86a6ed-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "402d7ca3-45e0-46c7-a954-050a4f86a6ed" (UID: "402d7ca3-45e0-46c7-a954-050a4f86a6ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:37:11.749677 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:11.749657    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deb9c86a-2b44-474f-8edd-a7e0183e137b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "deb9c86a-2b44-474f-8edd-a7e0183e137b" (UID: "deb9c86a-2b44-474f-8edd-a7e0183e137b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:37:11.751541 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:11.751506    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/402d7ca3-45e0-46c7-a954-050a4f86a6ed-kube-api-access-r5qs4" (OuterVolumeSpecName: "kube-api-access-r5qs4") pod "402d7ca3-45e0-46c7-a954-050a4f86a6ed" (UID: "402d7ca3-45e0-46c7-a954-050a4f86a6ed"). InnerVolumeSpecName "kube-api-access-r5qs4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:37:11.751839 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:11.751820    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01094640-b811-412f-b9bb-2d3cc7900efa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "01094640-b811-412f-b9bb-2d3cc7900efa" (UID: "01094640-b811-412f-b9bb-2d3cc7900efa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:37:11.752021 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:11.751998    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01094640-b811-412f-b9bb-2d3cc7900efa-kube-api-access-5nr68" (OuterVolumeSpecName: "kube-api-access-5nr68") pod "01094640-b811-412f-b9bb-2d3cc7900efa" (UID: "01094640-b811-412f-b9bb-2d3cc7900efa"). InnerVolumeSpecName "kube-api-access-5nr68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:37:11.752129 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:11.752020    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb9c86a-2b44-474f-8edd-a7e0183e137b-kube-api-access-nwxc2" (OuterVolumeSpecName: "kube-api-access-nwxc2") pod "deb9c86a-2b44-474f-8edd-a7e0183e137b" (UID: "deb9c86a-2b44-474f-8edd-a7e0183e137b"). InnerVolumeSpecName "kube-api-access-nwxc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:37:11.850244 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:11.850199    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nr68\" (UniqueName: \"kubernetes.io/projected/01094640-b811-412f-b9bb-2d3cc7900efa-kube-api-access-5nr68\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:37:11.850244 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:11.850240    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwxc2\" (UniqueName: \"kubernetes.io/projected/deb9c86a-2b44-474f-8edd-a7e0183e137b-kube-api-access-nwxc2\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:37:11.850244 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:11.850252    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01094640-b811-412f-b9bb-2d3cc7900efa-maas-api-tls\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:37:11.850244 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:11.850260    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01094640-b811-412f-b9bb-2d3cc7900efa-openshift-service-ca\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:37:11.850513 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:11.850269    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/402d7ca3-45e0-46c7-a954-050a4f86a6ed-plugins-config-volume\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:37:11.850513 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:11.850285    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/deb9c86a-2b44-474f-8edd-a7e0183e137b-plugins-config-volume\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:37:11.850513 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:11.850294    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5qs4\" (UniqueName: \"kubernetes.io/projected/402d7ca3-45e0-46c7-a954-050a4f86a6ed-kube-api-access-r5qs4\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:37:12.391654 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:12.391614    2563 generic.go:356] "Generic (PLEG): container finished" podID="01094640-b811-412f-b9bb-2d3cc7900efa" containerID="93e31cd51ab19ee6bc05b590cd6643d28811ffbd4e0b20ad0122622f0fd9a7b8" exitCode=0
Jul 24 17:37:12.392139 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:12.391678    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-de68c7-79ccd89d4f-sxqrz"
Jul 24 17:37:12.392139 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:12.391698    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-de68c7-79ccd89d4f-sxqrz" event={"ID":"01094640-b811-412f-b9bb-2d3cc7900efa","Type":"ContainerDied","Data":"93e31cd51ab19ee6bc05b590cd6643d28811ffbd4e0b20ad0122622f0fd9a7b8"}
Jul 24 17:37:12.392139 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:12.391771    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-de68c7-79ccd89d4f-sxqrz" event={"ID":"01094640-b811-412f-b9bb-2d3cc7900efa","Type":"ContainerDied","Data":"7c01c49358557cd16442e7379c3c770b67feca3e353cba254ef7df924c142526"}
Jul 24 17:37:12.392139 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:12.391800    2563 scope.go:117] "RemoveContainer" containerID="93e31cd51ab19ee6bc05b590cd6643d28811ffbd4e0b20ad0122622f0fd9a7b8"
Jul 24 17:37:12.393177 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:12.393152    2563 generic.go:356] "Generic (PLEG): container finished" podID="deb9c86a-2b44-474f-8edd-a7e0183e137b" containerID="7788c7de6b943929be64e3c9f1b8b7f62a1989506050cc8767333be4fff5d301" exitCode=0
Jul 24 17:37:12.393259 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:12.393215    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-de68c7-676cb7d874-q76wx"
Jul 24 17:37:12.393259 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:12.393236    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-de68c7-676cb7d874-q76wx" event={"ID":"deb9c86a-2b44-474f-8edd-a7e0183e137b","Type":"ContainerDied","Data":"7788c7de6b943929be64e3c9f1b8b7f62a1989506050cc8767333be4fff5d301"}
Jul 24 17:37:12.393371 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:12.393286    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-de68c7-676cb7d874-q76wx" event={"ID":"deb9c86a-2b44-474f-8edd-a7e0183e137b","Type":"ContainerDied","Data":"3acae923a5f98b2de183e7f2527c37b885f7beb68610c3befda18c6d79c4c61f"}
Jul 24 17:37:12.394658 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:12.394637    2563 generic.go:356] "Generic (PLEG): container finished" podID="402d7ca3-45e0-46c7-a954-050a4f86a6ed" containerID="3394a5c8058c6a0d6a205c59aba3fb4cfb94e90748a7eb66104d5100597d4bd9" exitCode=0
Jul 24 17:37:12.394736 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:12.394685    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-de68c7-7b7c5fd57b-dj4g8"
Jul 24 17:37:12.394736 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:12.394692    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-de68c7-7b7c5fd57b-dj4g8" event={"ID":"402d7ca3-45e0-46c7-a954-050a4f86a6ed","Type":"ContainerDied","Data":"3394a5c8058c6a0d6a205c59aba3fb4cfb94e90748a7eb66104d5100597d4bd9"}
Jul 24 17:37:12.394824 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:12.394734    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-de68c7-7b7c5fd57b-dj4g8" event={"ID":"402d7ca3-45e0-46c7-a954-050a4f86a6ed","Type":"ContainerDied","Data":"36e46209db3c9d9f34062ddd40790b6a4a1220b5527b71cd7dad35c28fa0dadb"}
Jul 24 17:37:12.403974 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:12.403913    2563 scope.go:117] "RemoveContainer" containerID="93e31cd51ab19ee6bc05b590cd6643d28811ffbd4e0b20ad0122622f0fd9a7b8"
Jul 24 17:37:12.404302 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:37:12.404271    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93e31cd51ab19ee6bc05b590cd6643d28811ffbd4e0b20ad0122622f0fd9a7b8\": container with ID starting with 93e31cd51ab19ee6bc05b590cd6643d28811ffbd4e0b20ad0122622f0fd9a7b8 not found: ID does not exist" containerID="93e31cd51ab19ee6bc05b590cd6643d28811ffbd4e0b20ad0122622f0fd9a7b8"
Jul 24 17:37:12.404394 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:12.404306    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93e31cd51ab19ee6bc05b590cd6643d28811ffbd4e0b20ad0122622f0fd9a7b8"} err="failed to get container status \"93e31cd51ab19ee6bc05b590cd6643d28811ffbd4e0b20ad0122622f0fd9a7b8\": rpc error: code = NotFound desc = could not find container \"93e31cd51ab19ee6bc05b590cd6643d28811ffbd4e0b20ad0122622f0fd9a7b8\": container with ID starting with 93e31cd51ab19ee6bc05b590cd6643d28811ffbd4e0b20ad0122622f0fd9a7b8 not found: ID does not exist"
Jul 24 17:37:12.404456 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:12.404396    2563 scope.go:117] "RemoveContainer" containerID="7788c7de6b943929be64e3c9f1b8b7f62a1989506050cc8767333be4fff5d301"
Jul 24 17:37:12.412695 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:12.412676    2563 scope.go:117] "RemoveContainer" containerID="7788c7de6b943929be64e3c9f1b8b7f62a1989506050cc8767333be4fff5d301"
Jul 24 17:37:12.413020 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:37:12.412997    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7788c7de6b943929be64e3c9f1b8b7f62a1989506050cc8767333be4fff5d301\": container with ID starting with 7788c7de6b943929be64e3c9f1b8b7f62a1989506050cc8767333be4fff5d301 not found: ID does not exist" containerID="7788c7de6b943929be64e3c9f1b8b7f62a1989506050cc8767333be4fff5d301"
Jul 24 17:37:12.413130 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:12.413029    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7788c7de6b943929be64e3c9f1b8b7f62a1989506050cc8767333be4fff5d301"} err="failed to get container status \"7788c7de6b943929be64e3c9f1b8b7f62a1989506050cc8767333be4fff5d301\": rpc error: code = NotFound desc = could not find container \"7788c7de6b943929be64e3c9f1b8b7f62a1989506050cc8767333be4fff5d301\": container with ID starting with 7788c7de6b943929be64e3c9f1b8b7f62a1989506050cc8767333be4fff5d301 not found: ID does not exist"
Jul 24 17:37:12.413130 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:12.413051    2563 scope.go:117] "RemoveContainer" containerID="3394a5c8058c6a0d6a205c59aba3fb4cfb94e90748a7eb66104d5100597d4bd9"
Jul 24 17:37:12.415100 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:12.415080    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-de68c7-79ccd89d4f-sxqrz"]
Jul 24 17:37:12.418821 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:12.418797    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-de68c7-79ccd89d4f-sxqrz"]
Jul 24 17:37:12.421036 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:12.421020    2563 scope.go:117] "RemoveContainer" containerID="3394a5c8058c6a0d6a205c59aba3fb4cfb94e90748a7eb66104d5100597d4bd9"
Jul 24 17:37:12.421262 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:37:12.421247    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3394a5c8058c6a0d6a205c59aba3fb4cfb94e90748a7eb66104d5100597d4bd9\": container with ID starting with 3394a5c8058c6a0d6a205c59aba3fb4cfb94e90748a7eb66104d5100597d4bd9 not found: ID does not exist" containerID="3394a5c8058c6a0d6a205c59aba3fb4cfb94e90748a7eb66104d5100597d4bd9"
Jul 24 17:37:12.421320 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:12.421269    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3394a5c8058c6a0d6a205c59aba3fb4cfb94e90748a7eb66104d5100597d4bd9"} err="failed to get container status \"3394a5c8058c6a0d6a205c59aba3fb4cfb94e90748a7eb66104d5100597d4bd9\": rpc error: code = NotFound desc = could not find container \"3394a5c8058c6a0d6a205c59aba3fb4cfb94e90748a7eb66104d5100597d4bd9\": container with ID starting with 3394a5c8058c6a0d6a205c59aba3fb4cfb94e90748a7eb66104d5100597d4bd9 not found: ID does not exist"
Jul 24 17:37:12.429906 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:12.429882    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-de68c7-676cb7d874-q76wx"]
Jul 24 17:37:12.433236 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:12.433215    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-de68c7-676cb7d874-q76wx"]
Jul 24 17:37:12.443558 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:12.443536    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-de68c7-7b7c5fd57b-dj4g8"]
Jul 24 17:37:12.447921 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:12.447902    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-de68c7-7b7c5fd57b-dj4g8"]
Jul 24 17:37:12.624716 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:12.624633    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:12.684955 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:12.684899    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:14.002421 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:14.002381    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01094640-b811-412f-b9bb-2d3cc7900efa" path="/var/lib/kubelet/pods/01094640-b811-412f-b9bb-2d3cc7900efa/volumes"
Jul 24 17:37:14.002806 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:14.002791    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="402d7ca3-45e0-46c7-a954-050a4f86a6ed" path="/var/lib/kubelet/pods/402d7ca3-45e0-46c7-a954-050a4f86a6ed/volumes"
Jul 24 17:37:14.003177 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:14.003163    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="deb9c86a-2b44-474f-8edd-a7e0183e137b" path="/var/lib/kubelet/pods/deb9c86a-2b44-474f-8edd-a7e0183e137b/volumes"
Jul 24 17:37:15.821423 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:15.821393    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:16.219581 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:16.219548    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:22.226145 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:22.226101    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:22.327247 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:22.327213    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:22.499109 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:22.499012    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:22.730247 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:22.730210    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:23.921782 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:23.921742    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:24.030599 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:24.030562    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:33.434622 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:33.434588    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d73e95-7f4cdbc78-5x8f9"]
Jul 24 17:37:33.435140 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:33.434800    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d73e95-7f4cdbc78-5x8f9" podUID="19014f4d-b299-4de5-93bd-a1c030ad60c7" containerName="maas-api" containerID="cri-o://2ea25235b7a32c791c7bad00f425a87b8bf5ae3a213ea13fdd676cdf9fbcfde4" gracePeriod=30
Jul 24 17:37:33.462702 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:33.462672    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:33.507278 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:33.507239    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-d73e95-5b677fb66b-254qn"]
Jul 24 17:37:33.507533 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:33.507482    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-d73e95-5b677fb66b-254qn" podUID="0c04556e-ad7a-431c-b4a9-a9372f613cfa" containerName="payload-processing" containerID="cri-o://cc25f31bc043c0b455eb75699367e3bcacd8a6e50c1cdeddc78ef89a4232e4ef" gracePeriod=30
Jul 24 17:37:33.512242 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:33.512213    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-d73e95-579c4f64d-cpqn6"]
Jul 24 17:37:33.512695 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:33.512640    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d73e95-579c4f64d-cpqn6" podUID="65bebe11-06f7-4960-ab05-3cdcfb8432c2" containerName="payload-pre-processing" containerID="cri-o://8feb4b13231552c839c84b0192bf5b6896475cad2a733236fa4640759bbb14e3" gracePeriod=30
Jul 24 17:37:33.554693 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:33.554662    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:33.622187 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:33.622153    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:33.791250 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:33.791220    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-d73e95-5b677fb66b-254qn"
Jul 24 17:37:33.797259 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:33.797235    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d73e95-7f4cdbc78-5x8f9"
Jul 24 17:37:33.807404 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:33.807383    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d73e95-579c4f64d-cpqn6"
Jul 24 17:37:33.854031 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:33.853995    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c04556e-ad7a-431c-b4a9-a9372f613cfa-plugins-config-volume\") pod \"0c04556e-ad7a-431c-b4a9-a9372f613cfa\" (UID: \"0c04556e-ad7a-431c-b4a9-a9372f613cfa\") "
Jul 24 17:37:33.854207 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:33.854081    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8rcp\" (UniqueName: \"kubernetes.io/projected/0c04556e-ad7a-431c-b4a9-a9372f613cfa-kube-api-access-n8rcp\") pod \"0c04556e-ad7a-431c-b4a9-a9372f613cfa\" (UID: \"0c04556e-ad7a-431c-b4a9-a9372f613cfa\") "
Jul 24 17:37:33.854415 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:33.854383    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c04556e-ad7a-431c-b4a9-a9372f613cfa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0c04556e-ad7a-431c-b4a9-a9372f613cfa" (UID: "0c04556e-ad7a-431c-b4a9-a9372f613cfa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:37:33.856131 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:33.856108    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c04556e-ad7a-431c-b4a9-a9372f613cfa-kube-api-access-n8rcp" (OuterVolumeSpecName: "kube-api-access-n8rcp") pod "0c04556e-ad7a-431c-b4a9-a9372f613cfa" (UID: "0c04556e-ad7a-431c-b4a9-a9372f613cfa"). InnerVolumeSpecName "kube-api-access-n8rcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:37:33.954800 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:33.954765    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19014f4d-b299-4de5-93bd-a1c030ad60c7-openshift-service-ca\") pod \"19014f4d-b299-4de5-93bd-a1c030ad60c7\" (UID: \"19014f4d-b299-4de5-93bd-a1c030ad60c7\") "
Jul 24 17:37:33.954800 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:33.954802    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7v8h\" (UniqueName: \"kubernetes.io/projected/19014f4d-b299-4de5-93bd-a1c030ad60c7-kube-api-access-h7v8h\") pod \"19014f4d-b299-4de5-93bd-a1c030ad60c7\" (UID: \"19014f4d-b299-4de5-93bd-a1c030ad60c7\") "
Jul 24 17:37:33.955108 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:33.954870    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65bebe11-06f7-4960-ab05-3cdcfb8432c2-plugins-config-volume\") pod \"65bebe11-06f7-4960-ab05-3cdcfb8432c2\" (UID: \"65bebe11-06f7-4960-ab05-3cdcfb8432c2\") "
Jul 24 17:37:33.955108 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:33.954999    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19014f4d-b299-4de5-93bd-a1c030ad60c7-maas-api-tls\") pod \"19014f4d-b299-4de5-93bd-a1c030ad60c7\" (UID: \"19014f4d-b299-4de5-93bd-a1c030ad60c7\") "
Jul 24 17:37:33.955108 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:33.955101    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hdgc\" (UniqueName: \"kubernetes.io/projected/65bebe11-06f7-4960-ab05-3cdcfb8432c2-kube-api-access-7hdgc\") pod \"65bebe11-06f7-4960-ab05-3cdcfb8432c2\" (UID: \"65bebe11-06f7-4960-ab05-3cdcfb8432c2\") "
Jul 24 17:37:33.955273 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:33.955219    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19014f4d-b299-4de5-93bd-a1c030ad60c7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "19014f4d-b299-4de5-93bd-a1c030ad60c7" (UID: "19014f4d-b299-4de5-93bd-a1c030ad60c7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:37:33.955273 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:33.955239    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65bebe11-06f7-4960-ab05-3cdcfb8432c2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "65bebe11-06f7-4960-ab05-3cdcfb8432c2" (UID: "65bebe11-06f7-4960-ab05-3cdcfb8432c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:37:33.955448 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:33.955423    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65bebe11-06f7-4960-ab05-3cdcfb8432c2-plugins-config-volume\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:37:33.955448 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:33.955446    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c04556e-ad7a-431c-b4a9-a9372f613cfa-plugins-config-volume\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:37:33.955637 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:33.955456    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8rcp\" (UniqueName: \"kubernetes.io/projected/0c04556e-ad7a-431c-b4a9-a9372f613cfa-kube-api-access-n8rcp\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:37:33.955637 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:33.955466    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19014f4d-b299-4de5-93bd-a1c030ad60c7-openshift-service-ca\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:37:33.957104 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:33.957082    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19014f4d-b299-4de5-93bd-a1c030ad60c7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "19014f4d-b299-4de5-93bd-a1c030ad60c7" (UID: "19014f4d-b299-4de5-93bd-a1c030ad60c7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:37:33.957193 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:33.957099    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65bebe11-06f7-4960-ab05-3cdcfb8432c2-kube-api-access-7hdgc" (OuterVolumeSpecName: "kube-api-access-7hdgc") pod "65bebe11-06f7-4960-ab05-3cdcfb8432c2" (UID: "65bebe11-06f7-4960-ab05-3cdcfb8432c2"). InnerVolumeSpecName "kube-api-access-7hdgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:37:33.957193 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:33.957181    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19014f4d-b299-4de5-93bd-a1c030ad60c7-kube-api-access-h7v8h" (OuterVolumeSpecName: "kube-api-access-h7v8h") pod "19014f4d-b299-4de5-93bd-a1c030ad60c7" (UID: "19014f4d-b299-4de5-93bd-a1c030ad60c7"). InnerVolumeSpecName "kube-api-access-h7v8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:37:34.056135 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:34.056096    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19014f4d-b299-4de5-93bd-a1c030ad60c7-maas-api-tls\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:37:34.056135 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:34.056129    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hdgc\" (UniqueName: \"kubernetes.io/projected/65bebe11-06f7-4960-ab05-3cdcfb8432c2-kube-api-access-7hdgc\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:37:34.056135 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:34.056141    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7v8h\" (UniqueName: \"kubernetes.io/projected/19014f4d-b299-4de5-93bd-a1c030ad60c7-kube-api-access-h7v8h\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\""
Jul 24 17:37:34.520580 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:34.520542    2563 generic.go:356] "Generic (PLEG): container finished" podID="65bebe11-06f7-4960-ab05-3cdcfb8432c2" containerID="8feb4b13231552c839c84b0192bf5b6896475cad2a733236fa4640759bbb14e3" exitCode=0
Jul 24 17:37:34.521073 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:34.520602    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d73e95-579c4f64d-cpqn6"
Jul 24 17:37:34.521073 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:34.520625    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d73e95-579c4f64d-cpqn6" event={"ID":"65bebe11-06f7-4960-ab05-3cdcfb8432c2","Type":"ContainerDied","Data":"8feb4b13231552c839c84b0192bf5b6896475cad2a733236fa4640759bbb14e3"}
Jul 24 17:37:34.521073 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:34.520704    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d73e95-579c4f64d-cpqn6" event={"ID":"65bebe11-06f7-4960-ab05-3cdcfb8432c2","Type":"ContainerDied","Data":"6f5697f450e69332f4ebbb97eeae2eb7ac794d52a6e53d3a87f831530c762406"}
Jul 24 17:37:34.521073 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:34.520717    2563 scope.go:117] "RemoveContainer" containerID="8feb4b13231552c839c84b0192bf5b6896475cad2a733236fa4640759bbb14e3"
Jul 24 17:37:34.522117 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:34.522031    2563 generic.go:356] "Generic (PLEG): container finished" podID="19014f4d-b299-4de5-93bd-a1c030ad60c7" containerID="2ea25235b7a32c791c7bad00f425a87b8bf5ae3a213ea13fdd676cdf9fbcfde4" exitCode=0
Jul 24 17:37:34.522117 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:34.522078    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d73e95-7f4cdbc78-5x8f9" event={"ID":"19014f4d-b299-4de5-93bd-a1c030ad60c7","Type":"ContainerDied","Data":"2ea25235b7a32c791c7bad00f425a87b8bf5ae3a213ea13fdd676cdf9fbcfde4"}
Jul 24 17:37:34.522117 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:34.522095    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d73e95-7f4cdbc78-5x8f9"
Jul 24 17:37:34.522344 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:34.522125    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d73e95-7f4cdbc78-5x8f9" event={"ID":"19014f4d-b299-4de5-93bd-a1c030ad60c7","Type":"ContainerDied","Data":"2c4465889a8afdece48038299dc3fa72e1827dae7eb73f648d27035459f6630b"}
Jul 24 17:37:34.523523 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:34.523504    2563 generic.go:356] "Generic (PLEG): container finished" podID="0c04556e-ad7a-431c-b4a9-a9372f613cfa" containerID="cc25f31bc043c0b455eb75699367e3bcacd8a6e50c1cdeddc78ef89a4232e4ef" exitCode=0
Jul 24 17:37:34.523637 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:34.523534    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-d73e95-5b677fb66b-254qn" event={"ID":"0c04556e-ad7a-431c-b4a9-a9372f613cfa","Type":"ContainerDied","Data":"cc25f31bc043c0b455eb75699367e3bcacd8a6e50c1cdeddc78ef89a4232e4ef"}
Jul 24 17:37:34.523637 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:34.523560    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-d73e95-5b677fb66b-254qn" event={"ID":"0c04556e-ad7a-431c-b4a9-a9372f613cfa","Type":"ContainerDied","Data":"f770d951e08a3f688bde59409a14490a3edacad2c67d3e1da762e4f0eb88cdda"}
Jul 24 17:37:34.523637 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:34.523568    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-d73e95-5b677fb66b-254qn"
Jul 24 17:37:34.530705 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:34.530687    2563 scope.go:117] "RemoveContainer" containerID="8feb4b13231552c839c84b0192bf5b6896475cad2a733236fa4640759bbb14e3"
Jul 24 17:37:34.531018 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:37:34.530997    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8feb4b13231552c839c84b0192bf5b6896475cad2a733236fa4640759bbb14e3\": container with ID starting with 8feb4b13231552c839c84b0192bf5b6896475cad2a733236fa4640759bbb14e3 not found: ID does not exist" containerID="8feb4b13231552c839c84b0192bf5b6896475cad2a733236fa4640759bbb14e3"
Jul 24 17:37:34.531060 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:34.531025    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8feb4b13231552c839c84b0192bf5b6896475cad2a733236fa4640759bbb14e3"} err="failed to get container status \"8feb4b13231552c839c84b0192bf5b6896475cad2a733236fa4640759bbb14e3\": rpc error: code = NotFound desc = could not find container \"8feb4b13231552c839c84b0192bf5b6896475cad2a733236fa4640759bbb14e3\": container with ID starting with 8feb4b13231552c839c84b0192bf5b6896475cad2a733236fa4640759bbb14e3 not found: ID does not exist"
Jul 24 17:37:34.531060 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:34.531043    2563 scope.go:117] "RemoveContainer" containerID="2ea25235b7a32c791c7bad00f425a87b8bf5ae3a213ea13fdd676cdf9fbcfde4"
Jul 24 17:37:34.539228 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:34.539208    2563 scope.go:117] "RemoveContainer" containerID="2ea25235b7a32c791c7bad00f425a87b8bf5ae3a213ea13fdd676cdf9fbcfde4"
Jul 24 17:37:34.539495 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:37:34.539474    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ea25235b7a32c791c7bad00f425a87b8bf5ae3a213ea13fdd676cdf9fbcfde4\": container with ID starting with 2ea25235b7a32c791c7bad00f425a87b8bf5ae3a213ea13fdd676cdf9fbcfde4 not found: ID does not exist" containerID="2ea25235b7a32c791c7bad00f425a87b8bf5ae3a213ea13fdd676cdf9fbcfde4"
Jul 24 17:37:34.539542 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:34.539503    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ea25235b7a32c791c7bad00f425a87b8bf5ae3a213ea13fdd676cdf9fbcfde4"} err="failed to get container status \"2ea25235b7a32c791c7bad00f425a87b8bf5ae3a213ea13fdd676cdf9fbcfde4\": rpc error: code = NotFound desc = could not find container \"2ea25235b7a32c791c7bad00f425a87b8bf5ae3a213ea13fdd676cdf9fbcfde4\": container with ID starting with 2ea25235b7a32c791c7bad00f425a87b8bf5ae3a213ea13fdd676cdf9fbcfde4 not found: ID does not exist"
Jul 24 17:37:34.539542 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:34.539519    2563 scope.go:117] "RemoveContainer" containerID="cc25f31bc043c0b455eb75699367e3bcacd8a6e50c1cdeddc78ef89a4232e4ef"
Jul 24 17:37:34.545111 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:34.545087    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-d73e95-579c4f64d-cpqn6"]
Jul 24 17:37:34.547843 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:34.547826    2563 scope.go:117] "RemoveContainer" containerID="cc25f31bc043c0b455eb75699367e3bcacd8a6e50c1cdeddc78ef89a4232e4ef"
Jul 24 17:37:34.548147 ip-10-0-133-234 kubenswrapper[2563]: E0724 17:37:34.548125    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc25f31bc043c0b455eb75699367e3bcacd8a6e50c1cdeddc78ef89a4232e4ef\": container with ID starting with cc25f31bc043c0b455eb75699367e3bcacd8a6e50c1cdeddc78ef89a4232e4ef not found: ID does not exist" containerID="cc25f31bc043c0b455eb75699367e3bcacd8a6e50c1cdeddc78ef89a4232e4ef"
Jul 24 17:37:34.548251 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:34.548152    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc25f31bc043c0b455eb75699367e3bcacd8a6e50c1cdeddc78ef89a4232e4ef"} err="failed to get container status \"cc25f31bc043c0b455eb75699367e3bcacd8a6e50c1cdeddc78ef89a4232e4ef\": rpc error: code = NotFound desc = could not find container \"cc25f31bc043c0b455eb75699367e3bcacd8a6e50c1cdeddc78ef89a4232e4ef\": container with ID starting with cc25f31bc043c0b455eb75699367e3bcacd8a6e50c1cdeddc78ef89a4232e4ef not found: ID does not exist"
Jul 24 17:37:34.550866 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:34.550842    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-d73e95-579c4f64d-cpqn6"]
Jul 24 17:37:34.561886 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:34.561853    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-d73e95-5b677fb66b-254qn"]
Jul 24 17:37:34.567356 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:34.567326    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-d73e95-5b677fb66b-254qn"]
Jul 24 17:37:34.577823 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:34.577798    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d73e95-7f4cdbc78-5x8f9"]
Jul 24 17:37:34.582083 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:34.582062    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d73e95-7f4cdbc78-5x8f9"]
Jul 24 17:37:36.002024 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:36.001990    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c04556e-ad7a-431c-b4a9-a9372f613cfa" path="/var/lib/kubelet/pods/0c04556e-ad7a-431c-b4a9-a9372f613cfa/volumes"
Jul 24 17:37:36.002470 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:36.002389    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19014f4d-b299-4de5-93bd-a1c030ad60c7" path="/var/lib/kubelet/pods/19014f4d-b299-4de5-93bd-a1c030ad60c7/volumes"
Jul 24 17:37:36.002710 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:36.002696    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65bebe11-06f7-4960-ab05-3cdcfb8432c2" path="/var/lib/kubelet/pods/65bebe11-06f7-4960-ab05-3cdcfb8432c2/volumes"
Jul 24 17:37:37.934859 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:37.934825    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:38.339470 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:38.339377    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:40.155811 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:40.155781    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:44.452004 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:44.451962    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:44.561024 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:44.560991    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:44.819992 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:44.819888    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:45.014178 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:45.014145    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:51.116458 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:51.116426    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7d8c754f6b-qcxmb_ce02e79c-6549-4c69-b469-bba26b0567cd/manager/0.log"
Jul 24 17:37:51.475593 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:51.475565    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-769b58b787-964jz_a6782019-c847-41fb-94bb-b6f195b3ade3/copy-manifests/0.log"
Jul 24 17:37:51.488150 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:51.488127    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-769b58b787-964jz_a6782019-c847-41fb-94bb-b6f195b3ade3/manager/0.log"
Jul 24 17:37:52.095926 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:52.095895    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-67c5c4478-779b7_497455f7-9ae1-499a-be43-e74bb3cc5fc3/maas-api/0.log"
Jul 24 17:37:52.214642 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:52.214611    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29748540-7s96c_44441fd3-52da-4fe3-975c-60dd4af20404/cleanup/0.log"
Jul 24 17:37:52.558192 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:52.558163    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-5qhkt_b017af45-b099-4a44-877d-ce1b1530c07b/postgres/0.log"
Jul 24 17:37:55.212720 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:55.212690    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-ws5gb_58b983a6-8913-4a3c-814b-2ede89bc1629/discovery/0.log"
Jul 24 17:37:55.709916 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:55.709875    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-jgkzv_b2a197f9-b38b-4ac6-98d0-d361f04dfe9b/payload-pre-processing/0.log"
Jul 24 17:37:55.827999 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:55.827951    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-npzx6_669adb99-c557-4f65-95e0-af083882951f/payload-processing/0.log"
Jul 24 17:37:55.938776 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:55.938741    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-746d84cf85-s4sxm_f3934ff0-8fae-4d73-82e3-96276d9ff5b0/router/0.log"
Jul 24 17:37:56.393026 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:56.393001    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-6lzp5_bf8bdd3a-7faa-4a02-934a-b25c232ba6fe/main/0.log"
Jul 24 17:37:56.504552 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:56.504523    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-m2x4g_d1633363-139f-4c13-a590-7fb2d8de28e5/main/0.log"
Jul 24 17:37:58.234116 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:58.234093    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ddklx_fb5a9838-1e05-4f86-b46c-760bd7aaa038/console-operator/1.log"
Jul 24 17:37:58.246463 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:58.246443    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ddklx_fb5a9838-1e05-4f86-b46c-760bd7aaa038/console-operator/1.log"
Jul 24 17:37:59.079725 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:59.079695    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:37:59.188602 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:37:59.188574    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:02.006072 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:02.006040    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:02.094677 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:02.094640    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log" limit=1024
Jul 24 17:38:04.836120 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:04.836089    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vzbnm_573b3eaf-dd83-456c-b7e0-35cb5aad6884/global-pull-secret-syncer/0.log"
Jul 24 17:38:04.927984 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:04.927957    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-45v2q_855016a9-43a6-4765-815e-9739917af352/konnectivity-agent/0.log"
Jul 24 17:38:05.040346 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:05.040323    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-234.ec2.internal_5737b582bc5c83055d2673f557e5384a/haproxy/0.log"
Jul 24 17:38:12.940452 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:12.940422    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-2flbb_6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657/kube-state-metrics/0.log"
Jul 24 17:38:12.971994 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:12.971969    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-2flbb_6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657/kube-rbac-proxy-main/0.log"
Jul 24 17:38:12.994457 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:12.994434    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-2flbb_6ae0d5ba-9ae1-44ec-ac8f-6c0ae405b657/kube-rbac-proxy-self/0.log"
Jul 24 17:38:13.088206 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:13.088182    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8fr8p_ab6c9a81-da59-4986-9ec6-74c5197d229a/node-exporter/0.log"
Jul 24 17:38:13.107336 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:13.107317    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8fr8p_ab6c9a81-da59-4986-9ec6-74c5197d229a/kube-rbac-proxy/0.log"
Jul 24 17:38:13.129498 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:13.129475    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8fr8p_ab6c9a81-da59-4986-9ec6-74c5197d229a/init-textfile/0.log"
Jul 24 17:38:13.325466 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:13.325394    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-kw8ln_6c73dc3e-5e79-4af4-8dfe-b8f78849e114/kube-rbac-proxy-main/0.log"
Jul 24 17:38:13.346330 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:13.346306    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-kw8ln_6c73dc3e-5e79-4af4-8dfe-b8f78849e114/kube-rbac-proxy-self/0.log"
Jul 24 17:38:13.368043 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:13.368023    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-kw8ln_6c73dc3e-5e79-4af4-8dfe-b8f78849e114/openshift-state-metrics/0.log"
Jul 24 17:38:13.602387 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:13.602308    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-k5tn8_c796c75a-b5a5-4157-ae11-7474151d0327/prometheus-operator/0.log"
Jul 24 17:38:13.620728 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:13.620708    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-k5tn8_c796c75a-b5a5-4157-ae11-7474151d0327/kube-rbac-proxy/0.log"
Jul 24 17:38:13.646172 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:13.646149    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-cdmqk_9e6dcdc5-d8ee-47ca-ad4a-e9262d634f1c/prometheus-operator-admission-webhook/0.log"
Jul 24 17:38:14.968038 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:14.968007    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-mt8vp_7960f9f4-815e-4b60-bd8c-2a0654a61ae6/networking-console-plugin/0.log"
Jul 24 17:38:15.272883 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.272801    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ffwf8/perf-node-gather-daemonset-smft6"]
Jul 24 17:38:15.273207 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.273192    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69a0d029-1031-4849-9060-ff5fb0e35065" containerName="payload-pre-processing"
Jul 24 17:38:15.273252 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.273209    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="69a0d029-1031-4849-9060-ff5fb0e35065" containerName="payload-pre-processing"
Jul 24 17:38:15.273252 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.273221    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65bebe11-06f7-4960-ab05-3cdcfb8432c2" containerName="payload-pre-processing"
Jul 24 17:38:15.273252 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.273227    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="65bebe11-06f7-4960-ab05-3cdcfb8432c2" containerName="payload-pre-processing"
Jul 24 17:38:15.273252 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.273234    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f288f4c-fda7-45ea-b4a7-dfb5f463816a" containerName="payload-processing"
Jul 24 17:38:15.273252 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.273239    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f288f4c-fda7-45ea-b4a7-dfb5f463816a" containerName="payload-processing"
Jul 24 17:38:15.273252 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.273247    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab1da480-9c5c-4176-9794-a449e32e7196" containerName="payload-processing"
Jul 24 17:38:15.273252 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.273253    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab1da480-9c5c-4176-9794-a449e32e7196" containerName="payload-processing"
Jul 24 17:38:15.273473 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.273264    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="402d7ca3-45e0-46c7-a954-050a4f86a6ed" containerName="payload-pre-processing"
Jul 24 17:38:15.273473 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.273271    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="402d7ca3-45e0-46c7-a954-050a4f86a6ed" containerName="payload-pre-processing"
Jul 24 17:38:15.273473 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.273277    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deb9c86a-2b44-474f-8edd-a7e0183e137b" containerName="payload-processing"
Jul 24 17:38:15.273473 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.273282    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="deb9c86a-2b44-474f-8edd-a7e0183e137b" containerName="payload-processing"
Jul 24 17:38:15.273473 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.273289    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01094640-b811-412f-b9bb-2d3cc7900efa" containerName="maas-api"
Jul 24 17:38:15.273473 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.273294    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="01094640-b811-412f-b9bb-2d3cc7900efa" containerName="maas-api"
Jul 24 17:38:15.273473 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.273299    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01d35db8-e624-4941-9212-78752dc2c387" containerName="payload-processing"
Jul 24 17:38:15.273473 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.273304    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="01d35db8-e624-4941-9212-78752dc2c387" containerName="payload-processing"
Jul 24 17:38:15.273473 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.273309    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9fecf19-4c51-4a7c-8036-e64822d7ba44" containerName="payload-pre-processing"
Jul 24 17:38:15.273473 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.273314    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9fecf19-4c51-4a7c-8036-e64822d7ba44" containerName="payload-pre-processing"
Jul 24 17:38:15.273473 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.273320    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ae73fdd-bb7a-4ecf-8fc1-f3da7b17c795" containerName="payload-pre-processing"
Jul 24 17:38:15.273473 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.273325    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ae73fdd-bb7a-4ecf-8fc1-f3da7b17c795" containerName="payload-pre-processing"
Jul 24 17:38:15.273473 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.273332    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19014f4d-b299-4de5-93bd-a1c030ad60c7" containerName="maas-api"
Jul 24 17:38:15.273473 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.273337    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="19014f4d-b299-4de5-93bd-a1c030ad60c7" containerName="maas-api"
Jul 24 17:38:15.273473 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.273345    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c04556e-ad7a-431c-b4a9-a9372f613cfa" containerName="payload-processing"
Jul 24 17:38:15.273473 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.273350    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c04556e-ad7a-431c-b4a9-a9372f613cfa" containerName="payload-processing"
Jul 24 17:38:15.273473 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.273421    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="69a0d029-1031-4849-9060-ff5fb0e35065" containerName="payload-pre-processing"
Jul 24 17:38:15.273473 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.273431    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c04556e-ad7a-431c-b4a9-a9372f613cfa" containerName="payload-processing"
Jul 24 17:38:15.273473 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.273438    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab1da480-9c5c-4176-9794-a449e32e7196" containerName="payload-processing"
Jul 24 17:38:15.273473 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.273443    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="deb9c86a-2b44-474f-8edd-a7e0183e137b" containerName="payload-processing"
Jul 24 17:38:15.273473 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.273449    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9fecf19-4c51-4a7c-8036-e64822d7ba44" containerName="payload-pre-processing"
Jul 24 17:38:15.273473 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.273454    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="01094640-b811-412f-b9bb-2d3cc7900efa" containerName="maas-api"
Jul 24 17:38:15.273473 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.273459    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f288f4c-fda7-45ea-b4a7-dfb5f463816a" containerName="payload-processing"
Jul 24 17:38:15.273473 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.273465    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="01d35db8-e624-4941-9212-78752dc2c387" containerName="payload-processing"
Jul 24 17:38:15.273473 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.273471    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="402d7ca3-45e0-46c7-a954-050a4f86a6ed" containerName="payload-pre-processing"
Jul 24 17:38:15.273473 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.273477    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="19014f4d-b299-4de5-93bd-a1c030ad60c7" containerName="maas-api"
Jul 24 17:38:15.273473 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.273482    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="65bebe11-06f7-4960-ab05-3cdcfb8432c2" containerName="payload-pre-processing"
Jul 24 17:38:15.274562 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.273488    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ae73fdd-bb7a-4ecf-8fc1-f3da7b17c795" containerName="payload-pre-processing"
Jul 24 17:38:15.276787 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.276772    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ffwf8/perf-node-gather-daemonset-smft6"
Jul 24 17:38:15.279150 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.279131    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ffwf8\"/\"kube-root-ca.crt\""
Jul 24 17:38:15.280124 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.280104    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ffwf8\"/\"default-dockercfg-5cl2f\""
Jul 24 17:38:15.280223 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.280130    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ffwf8\"/\"openshift-service-ca.crt\""
Jul 24 17:38:15.287514 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.287490    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ffwf8/perf-node-gather-daemonset-smft6"]
Jul 24 17:38:15.418440 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.418391    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/76e02618-c69c-4d34-8e77-74662193f2c7-lib-modules\") pod \"perf-node-gather-daemonset-smft6\" (UID: \"76e02618-c69c-4d34-8e77-74662193f2c7\") " pod="openshift-must-gather-ffwf8/perf-node-gather-daemonset-smft6"
Jul 24 17:38:15.418602 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.418456    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76e02618-c69c-4d34-8e77-74662193f2c7-sys\") pod \"perf-node-gather-daemonset-smft6\" (UID: \"76e02618-c69c-4d34-8e77-74662193f2c7\") " pod="openshift-must-gather-ffwf8/perf-node-gather-daemonset-smft6"
Jul 24 17:38:15.418602 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.418495    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/76e02618-c69c-4d34-8e77-74662193f2c7-proc\") pod \"perf-node-gather-daemonset-smft6\" (UID: \"76e02618-c69c-4d34-8e77-74662193f2c7\") " pod="openshift-must-gather-ffwf8/perf-node-gather-daemonset-smft6"
Jul 24 17:38:15.418602 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.418514    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77pmr\" (UniqueName: \"kubernetes.io/projected/76e02618-c69c-4d34-8e77-74662193f2c7-kube-api-access-77pmr\") pod \"perf-node-gather-daemonset-smft6\" (UID: \"76e02618-c69c-4d34-8e77-74662193f2c7\") " pod="openshift-must-gather-ffwf8/perf-node-gather-daemonset-smft6"
Jul 24 17:38:15.418716 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.418615    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/76e02618-c69c-4d34-8e77-74662193f2c7-podres\") pod \"perf-node-gather-daemonset-smft6\" (UID: \"76e02618-c69c-4d34-8e77-74662193f2c7\") " pod="openshift-must-gather-ffwf8/perf-node-gather-daemonset-smft6"
Jul 24 17:38:15.519238 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.519200    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77pmr\" (UniqueName: \"kubernetes.io/projected/76e02618-c69c-4d34-8e77-74662193f2c7-kube-api-access-77pmr\") pod \"perf-node-gather-daemonset-smft6\" (UID: \"76e02618-c69c-4d34-8e77-74662193f2c7\") " pod="openshift-must-gather-ffwf8/perf-node-gather-daemonset-smft6"
Jul 24 17:38:15.519399 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.519272    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/76e02618-c69c-4d34-8e77-74662193f2c7-podres\") pod \"perf-node-gather-daemonset-smft6\" (UID: \"76e02618-c69c-4d34-8e77-74662193f2c7\") " pod="openshift-must-gather-ffwf8/perf-node-gather-daemonset-smft6"
Jul 24 17:38:15.519399 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.519303    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/76e02618-c69c-4d34-8e77-74662193f2c7-lib-modules\") pod \"perf-node-gather-daemonset-smft6\" (UID: \"76e02618-c69c-4d34-8e77-74662193f2c7\") " pod="openshift-must-gather-ffwf8/perf-node-gather-daemonset-smft6"
Jul 24 17:38:15.519399 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.519360    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76e02618-c69c-4d34-8e77-74662193f2c7-sys\") pod \"perf-node-gather-daemonset-smft6\" (UID: \"76e02618-c69c-4d34-8e77-74662193f2c7\") " pod="openshift-must-gather-ffwf8/perf-node-gather-daemonset-smft6"
Jul 24 17:38:15.519540 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.519403    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/76e02618-c69c-4d34-8e77-74662193f2c7-proc\") pod \"perf-node-gather-daemonset-smft6\" (UID: \"76e02618-c69c-4d34-8e77-74662193f2c7\") " pod="openshift-must-gather-ffwf8/perf-node-gather-daemonset-smft6"
Jul 24 17:38:15.519540 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.519458    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/76e02618-c69c-4d34-8e77-74662193f2c7-podres\") pod \"perf-node-gather-daemonset-smft6\" (UID: \"76e02618-c69c-4d34-8e77-74662193f2c7\") " pod="openshift-must-gather-ffwf8/perf-node-gather-daemonset-smft6"
Jul 24 17:38:15.519540 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.519478    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76e02618-c69c-4d34-8e77-74662193f2c7-sys\") pod \"perf-node-gather-daemonset-smft6\" (UID: \"76e02618-c69c-4d34-8e77-74662193f2c7\") " pod="openshift-must-gather-ffwf8/perf-node-gather-daemonset-smft6"
Jul 24 17:38:15.519540 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.519479    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/76e02618-c69c-4d34-8e77-74662193f2c7-proc\") pod \"perf-node-gather-daemonset-smft6\" (UID: \"76e02618-c69c-4d34-8e77-74662193f2c7\") " pod="openshift-must-gather-ffwf8/perf-node-gather-daemonset-smft6"
Jul 24 17:38:15.519540 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.519501    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/76e02618-c69c-4d34-8e77-74662193f2c7-lib-modules\") pod \"perf-node-gather-daemonset-smft6\" (UID: \"76e02618-c69c-4d34-8e77-74662193f2c7\") " pod="openshift-must-gather-ffwf8/perf-node-gather-daemonset-smft6"
Jul 24 17:38:15.530458 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.530396    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77pmr\" (UniqueName: \"kubernetes.io/projected/76e02618-c69c-4d34-8e77-74662193f2c7-kube-api-access-77pmr\") pod \"perf-node-gather-daemonset-smft6\" (UID: \"76e02618-c69c-4d34-8e77-74662193f2c7\") " pod="openshift-must-gather-ffwf8/perf-node-gather-daemonset-smft6"
Jul 24 17:38:15.587073 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.587042    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ffwf8/perf-node-gather-daemonset-smft6"
Jul 24 17:38:15.622355 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.622319    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ddklx_fb5a9838-1e05-4f86-b46c-760bd7aaa038/console-operator/1.log"
Jul 24 17:38:15.627252 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.627226    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ddklx_fb5a9838-1e05-4f86-b46c-760bd7aaa038/console-operator/2.log"
Jul 24 17:38:15.718646 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.718619    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ffwf8/perf-node-gather-daemonset-smft6"]
Jul 24 17:38:15.720394 ip-10-0-133-234 kubenswrapper[2563]: W0724 17:38:15.720371    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod76e02618_c69c_4d34_8e77_74662193f2c7.slice/crio-12ef8b46cee79b77627b34b099eb019be72af300583629c084f8b7234ab6bae0 WatchSource:0}: Error finding container 12ef8b46cee79b77627b34b099eb019be72af300583629c084f8b7234ab6bae0: Status 404 returned error can't find the container with id 12ef8b46cee79b77627b34b099eb019be72af300583629c084f8b7234ab6bae0
Jul 24 17:38:15.727157 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:15.727112    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ffwf8/perf-node-gather-daemonset-smft6" event={"ID":"76e02618-c69c-4d34-8e77-74662193f2c7","Type":"ContainerStarted","Data":"12ef8b46cee79b77627b34b099eb019be72af300583629c084f8b7234ab6bae0"}
Jul 24 17:38:16.732725 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:16.732685    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ffwf8/perf-node-gather-daemonset-smft6" event={"ID":"76e02618-c69c-4d34-8e77-74662193f2c7","Type":"ContainerStarted","Data":"9cb8f86c9c49c9895119597253742b5ac2df2c4c50436bd7d17392372dcc457a"}
Jul 24 17:38:16.733172 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:16.732756    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ffwf8/perf-node-gather-daemonset-smft6"
Jul 24 17:38:16.750565 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:16.750518    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ffwf8/perf-node-gather-daemonset-smft6" podStartSLOduration=1.750503094 podStartE2EDuration="1.750503094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:38:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:38:16.748661209 +0000 UTC m=+3319.390625093" watchObservedRunningTime="2026-07-24 17:38:16.750503094 +0000 UTC m=+3319.392466979"
Jul 24 17:38:16.823728 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:16.823700    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-62xxb_1f6e96ed-11c0-4dbb-afb0-231d599a870b/volume-data-source-validator/0.log"
Jul 24 17:38:17.898362 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:17.898334    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vrscm_708abf59-5c2a-46e2-8506-386e91731078/dns/0.log"
Jul 24 17:38:17.917185 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:17.917162    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vrscm_708abf59-5c2a-46e2-8506-386e91731078/kube-rbac-proxy/0.log"
Jul 24 17:38:17.961696 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:17.961673    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g5sl6_f263630d-88c9-4b5d-8afb-7075a1118555/dns-node-resolver/0.log"
Jul 24 17:38:18.630523 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:18.630495    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vq4cx_ce9a1972-6a1e-4e0d-9dab-0e25820dedf1/node-ca/0.log"
Jul 24 17:38:19.717835 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:19.717803    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-ws5gb_58b983a6-8913-4a3c-814b-2ede89bc1629/discovery/0.log"
Jul 24 17:38:19.918556 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:19.918529    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-jgkzv_b2a197f9-b38b-4ac6-98d0-d361f04dfe9b/payload-pre-processing/0.log"
Jul 24 17:38:19.970552 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:19.970467    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-npzx6_669adb99-c557-4f65-95e0-af083882951f/payload-processing/0.log"
Jul 24 17:38:20.016331 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:20.016304    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-746d84cf85-s4sxm_f3934ff0-8fae-4d73-82e3-96276d9ff5b0/router/0.log"
Jul 24 17:38:20.869063 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:20.869033    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8zdrn_17291b08-516e-48fe-9b2a-e8db21ed7c6c/serve-healthcheck-canary/0.log"
Jul 24 17:38:21.387087 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:21.387055    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-5qj6l_ef1872b5-49d6-4162-bcba-afea4c9ca621/insights-operator/0.log"
Jul 24 17:38:21.387262 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:21.387127    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-5qj6l_ef1872b5-49d6-4162-bcba-afea4c9ca621/insights-operator/1.log"
Jul 24 17:38:21.550100 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:21.550071    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j7rx8_3c99ab05-563d-4bc8-882a-49defb5b97a9/kube-rbac-proxy/0.log"
Jul 24 17:38:21.572061 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:21.572041    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j7rx8_3c99ab05-563d-4bc8-882a-49defb5b97a9/exporter/0.log"
Jul 24 17:38:21.593608 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:21.593585    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j7rx8_3c99ab05-563d-4bc8-882a-49defb5b97a9/extractor/0.log"
Jul 24 17:38:22.749127 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:22.749097    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ffwf8/perf-node-gather-daemonset-smft6"
Jul 24 17:38:24.364357 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:24.364328    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7d8c754f6b-qcxmb_ce02e79c-6549-4c69-b469-bba26b0567cd/manager/0.log"
Jul 24 17:38:24.626498 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:24.626409    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-769b58b787-964jz_a6782019-c847-41fb-94bb-b6f195b3ade3/manager/0.log"
Jul 24 17:38:24.660474 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:24.660449    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-769b58b787-964jz_a6782019-c847-41fb-94bb-b6f195b3ade3/copy-manifests/0.log"
Jul 24 17:38:25.555892 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:25.555859    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-ws5gb_58b983a6-8913-4a3c-814b-2ede89bc1629/discovery/0.log"
Jul 24 17:38:25.742523 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:25.742491    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-jgkzv_b2a197f9-b38b-4ac6-98d0-d361f04dfe9b/payload-pre-processing/0.log"
Jul 24 17:38:25.762502 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:25.762480    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-npzx6_669adb99-c557-4f65-95e0-af083882951f/payload-processing/0.log"
Jul 24 17:38:25.775076 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:25.775059    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-746d84cf85-s4sxm_f3934ff0-8fae-4d73-82e3-96276d9ff5b0/router/0.log"
Jul 24 17:38:26.188143 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:26.188108    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-675dd78886-xvb6c_82bb361f-6db3-4b26-9d8b-117ae0929ad5/manager/0.log"
Jul 24 17:38:27.333158 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:27.333125    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-ws5gb_58b983a6-8913-4a3c-814b-2ede89bc1629/discovery/0.log"
Jul 24 17:38:27.488479 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:27.488404    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-jgkzv_b2a197f9-b38b-4ac6-98d0-d361f04dfe9b/payload-pre-processing/0.log"
Jul 24 17:38:27.508249 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:27.508228    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-npzx6_669adb99-c557-4f65-95e0-af083882951f/payload-processing/0.log"
Jul 24 17:38:27.522074 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:27.522053    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-746d84cf85-s4sxm_f3934ff0-8fae-4d73-82e3-96276d9ff5b0/router/0.log"
Jul 24 17:38:31.499922 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:31.499842    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-xl9vk_e3460f62-6389-4673-9e33-e4e5aa0d4d9a/kube-storage-version-migrator-operator/1.log"
Jul 24 17:38:31.500666 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:31.500647    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-xl9vk_e3460f62-6389-4673-9e33-e4e5aa0d4d9a/kube-storage-version-migrator-operator/0.log"
Jul 24 17:38:32.808335 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:32.808309    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t4xnt_6a0daaab-0ecb-4e33-bc7c-c53caef203f2/kube-multus-additional-cni-plugins/0.log"
Jul 24 17:38:32.831781 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:32.831758    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t4xnt_6a0daaab-0ecb-4e33-bc7c-c53caef203f2/egress-router-binary-copy/0.log"
Jul 24 17:38:32.850644 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:32.850622    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t4xnt_6a0daaab-0ecb-4e33-bc7c-c53caef203f2/cni-plugins/0.log"
Jul 24 17:38:32.872503 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:32.872482    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t4xnt_6a0daaab-0ecb-4e33-bc7c-c53caef203f2/bond-cni-plugin/0.log"
Jul 24 17:38:32.891634 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:32.891614    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t4xnt_6a0daaab-0ecb-4e33-bc7c-c53caef203f2/routeoverride-cni/0.log"
Jul 24 17:38:32.909916 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:32.909899    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t4xnt_6a0daaab-0ecb-4e33-bc7c-c53caef203f2/whereabouts-cni-bincopy/0.log"
Jul 24 17:38:32.929108 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:32.929091    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t4xnt_6a0daaab-0ecb-4e33-bc7c-c53caef203f2/whereabouts-cni/0.log"
Jul 24 17:38:32.970598 ip-10-0-133-234 kubenswrapper[2563]: I0724 17:38:32.970578    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dblpg_e620ec4e-9703-45c1-9080-d17039c0d2e8/kube-multus/0.log"